|According to what you post as your target |
"an operating system specific to a certain set of needs and designed for only a few bits of hardware (though rather complex ones which serve multiple functions)"
why do you want to reinvent the wheel? (lacking the required know-how)
I suggest you develop an application environment based on an existing OS to be choosen according to the wished complexity. An open source OS is better suited since it is free and its source code fully accessible to extend or modify its facilities.
A good starting point may be FreeDOS that can be tailored to support very effective special operation oriented interfaces and devices, minimizing design complexity.
That way you can restrict your efforts to application programming almost avoiding the complexity of system programming (that requires hardware know-how too).