JWCCommandSpawn: Cross-Platform Process Execution and Interactive Command Pipes

View JWCCommandSpawn on GitHub

Back to overview

Modern applications rarely operate in complete isolation.

They invoke compilers, scripts, media tools, version-control systems, converters, diagnostic utilities, machine-learning processes, and operating-system commands. They may need to provide input to those processes, capture output, distinguish ordinary output from errors, inspect an exit status, or maintain an interactive shell across a sequence of commands.

The operating systems beneath those applications do not provide one universal process model.

Linux and Windows differ in process creation, file descriptors, command-line parsing, executable discovery, shell behavior, pipe management, quoting rules, and termination semantics. Even within one operating system, launching an executable directly is not the same as passing a command through Bash, cmd.exe, PowerShell, or Python.

JWCCommandSpawn provides a reusable boundary around those differences.

It is a cross-platform native and managed command-spawning library for the NewAge family of projects. A C++17 shared library performs the platform-specific process and pipe operations, while a .NET wrapper exposes those capabilities through a convenient managed API.

Its purpose is broader than simply starting a subprocess.

JWCCommandSpawn models:

• which command environment should interpret an instruction;

• how arguments must be escaped for that environment;

• which process streams should be connected;

• how input should be written;

• how output should be read;

• when a process has completed;

• what exit status it returned;

• how a persistent shell can host multiple commands.

It turns external-process execution into a deliberate subsystem rather than a scattered collection of shell calls.

The Process Boundary as Infrastructure

Launching an external program often begins as a one-line requirement.

A project needs to run a tool, so it constructs a command string and invokes an operating-system API. That may be sufficient until the command contains spaces, quotes, redirection, environment variables, Unicode text, or shell syntax. It becomes more complicated when the application needs to collect standard output, send input, detect errors, or support another operating system.

Eventually, process execution becomes architecture.

JWCCommandSpawn gives that architecture a dedicated home.

The library currently supports:

• launching external commands;

• direct or shell-mediated execution;

• selecting a default shell;

• selecting Bash;

• Python-oriented invocation;

• configuring an explicit shell;

• connecting standard input;

• connecting standard output;

• connecting standard error;

• reading bytes, lines, or complete output;

• writing bytes, strings, or lines;

• checking whether output is available;

• closing selected pipes;

• waiting for process completion;

• retrieving the process return value;

• maintaining a persistent command pipe.

These facilities can be used from native C++, through the exported ABI, or from modern .NET applications through P/Invoke.

Native Implementation, Shared Public Contract

The native portion of JWCCommandSpawn is implemented as a C++17 shared library.

Its public CommandSpawn abstraction defines the common behavior expected from process implementations, while Linux- and Windows-specific source files provide the actual operating-system integration.

The build selects the appropriate platform implementation:

Common CommandSpawn interface
        ├── Linux process implementation
        └── Windows process implementation

This follows the same platform-conscious design found elsewhere in the ecosystem.

The common code does not pretend that process creation is identical on Linux and Windows. Instead, it establishes the operations a consumer should be able to perform and allows each platform to implement those operations according to its native rules.

The public interface includes methods for:

• discovering known shells;

• testing whether a shell is available;

• selecting a shell;

• selecting an escapement style;

• launching a command;

• reading and writing connected streams;

• flushing input;

• closing streams;

• joining the child process.

An exported procedural API mirrors this native object interface. The shared library can therefore preserve an object-oriented internal design while presenting a stable seam for managed code and other foreign-function consumers.

Built on JWCEssentials

JWCCommandSpawn depends directly on JWCEssentials.

The native build includes the shared JWCEssentials CMake integration, links against the native JWCEssentials library, and uses foundational structures such as utf8_string_struct. The managed project similarly references the staged JWCEssentials.net assembly.

This relationship is technically useful and architecturally revealing.

JWCEssentials provides:

• shared native and managed string structures;

• import and export conventions;

• workspace path configuration;

• library discovery;

• common build staging;

• cross-repository environment assumptions.

JWCCommandSpawn then applies those foundations to one focused responsibility: starting and communicating with external processes.

The progression now looks like this:

JWCEssentials
    Shared structures
    Build and staging conventions
    Platform-conscious native foundations
             ↓
JWCCommandSpawn
    Process creation
    Shell selection
    Stream communication
    Exit and lifetime management

Because both projects participate in the same NewAge workspace, their native and managed outputs can be staged into predictable locations and discovered consistently by consuming repositories.

Explicit Stream Selection

A spawned process commonly exposes three standard streams:

• standard input;

• standard output;

• standard error.

JWCCommandSpawn represents these streams through combinable pipe flags:

E_PIPE_NONE
E_PIPE_STDOUT
E_PIPE_STDERR
E_PIPE_STDIN

Because the flags are independent, a caller can request only the connections it needs.

A fire-and-forget command may require no connected streams. A utility invocation may need only standard output. A diagnostic operation may need both output and error channels. An interactive child process may require standard input and standard output together.

This explicit selection has several benefits.

It makes process intent visible at the call site. It avoids creating unnecessary communication channels. It also gives the platform implementation a clear contract about which handles or file descriptors must be configured before the child begins execution.

In managed code, the flags can be combined naturally:

CommandSpawn.E_PIPE.E_PIPE_STDIN |
CommandSpawn.E_PIPE.E_PIPE_STDOUT

The same model is preserved across the native and managed boundary.

Simple Execution for Common Cases

For ordinary command execution, the managed wrapper provides a compact Exec method.

using JWCCommandSpawn.net;

var spawn = new CommandSpawn();

string output = spawn.Exec(
    "echo hello from JWCCommandSpawn"
);

Console.WriteLine(output);

Internally, the wrapper launches the command with standard output connected, reads the stream to completion, waits for the process to exit, and returns the collected text.

This gives callers a convenient high-level path without removing access to the lower-level operations.

An overload accepts input that should be provided to the spawned command. The wrapper tracks the resulting execution status after joining the child process.

This dual-level design is valuable. Simple tasks remain simple, while more complex callers can manage the streams and process lifetime directly.

Process Lifetime Is Not Output Lifetime

External-process code is easy to get subtly wrong because several different events can occur at different times:

• the command is launched;

• the child begins producing output;

• one output stream reaches end-of-file;

• the child closes its input;

• the child process exits;

• the parent retrieves the exit status;

• native handles are released.

JWCCommandSpawn keeps these operations explicit.

ReadToEnd consumes the selected output stream. Join waits for the process and returns its status. ClosePipe closes selected communication channels. The native object’s destructor and exported destroy operation handle final cleanup.

This allows callers to reason separately about communication and completion.

That distinction becomes especially important when the child waits for input. A process may not terminate until its input stream has been closed. Conversely, output may still need to be drained before waiting, depending on the amount of data and the buffering behavior of the platform.

By exposing these operations rather than burying them completely, JWCCommandSpawn supports both convenience and careful process control.

Reading at the Right Level

Different subprocess interactions require different forms of reading.

JWCCommandSpawn supports:

• testing whether data is available;

• reading one byte;

• reading one line;

• reading to the end of a stream;

• reading all currently relevant data.

A binary-oriented protocol may need byte-level access. A compiler or shell often produces line-oriented output. A one-shot utility may be easiest to consume as one complete string.

The API does not force every process into the same reading pattern.

The selected stream is also explicit, allowing callers to distinguish standard output from standard error when both have been connected.

This matters for tools whose error channel carries warnings, diagnostics, progress information, or structured messages that should not be mixed blindly with ordinary output.

Writing to the Child Process

The input side follows a similarly layered model.

A caller can write:

• an individual byte;

• a string;

• a line with the appropriate line ending;

• buffered data followed by an explicit flush.

This supports more than commands that receive one fixed input value.

It can also support:

• interactive interpreters;

• shell sessions;

• command-driven native tools;

• converters that consume standard input;

• processes using simple text protocols;

• utilities that expect a sequence of commands.

The distinction between WriteString and WriteLine is important for interactive tools. Some child processes do not act until they receive a line terminator, while others treat the input as a raw byte stream.

Shells as First-Class Configuration

JWCCommandSpawn represents a shell through a small structure containing:

• a descriptive name;

• the shell executable;

• the switch used to pass a command.

This lets the library reason about the command environment rather than treating the shell as an invisible operating-system assumption.

The API can provide:

• the platform’s default shell;

• Bash;

• Python;

• an explicitly configured shell.

It can also test whether a particular shell appears to be available before attempting to use it.

A managed caller can select Bash like this:

var spawn = new CommandSpawn();

var bash = spawn.GetShell_Bash();

if (spawn.HasShell(ref bash))
{
    spawn.SetShell(bash);
}

Explicit shell selection improves portability because the caller can state what kind of syntax it intends to use.

A command containing Bash substitutions should not accidentally be sent to cmd.exe. A PowerShell command should not be escaped as if it were a POSIX shell argument. A Python-oriented instruction has a different execution model from either.

JWCCommandSpawn makes the interpreter part of the operation’s declared context.

Command Escapement Is Context-Dependent

One of the most difficult parts of reliable process execution is command-line escaping.

There is no single universal escaping algorithm.

A string may be interpreted by:

• a direct Windows process command line;

• a POSIX shell;

• cmd.exe;

• PowerShell;

• another explicitly configured shell;

• no shell at all.

Each context has its own rules for quotes, spaces, backslashes, substitutions, metacharacters, and nested command strings.

JWCCommandSpawn therefore defines explicit escapement styles:

Auto
None / Raw
PosixShell
WindowsCommandLine
CmdExe
PowerShell

The library can choose automatically where appropriate, or the caller can specify the exact context.

This is a major improvement over ad hoc quoting.

Adding quotation marks around an argument is not, by itself, a complete escaping strategy. The correct transformation depends on which parser will consume the resulting text and whether one command processor is invoking another.

By naming these contexts, JWCCommandSpawn acknowledges that command construction is a translation problem.

The source value belongs to the calling application. The escaped representation belongs to the target command environment.

Direct Execution and Shell Execution Are Different Tools

A shell is useful when a command needs shell features such as:

• pipelines;

• redirection;

• substitutions;

• wildcard expansion;

• compound statements;

• shell variables;

• built-in commands.

But a shell also introduces another parser and another layer of quoting.

Direct execution can be safer and more predictable when the application already knows the executable and its arguments. Shell execution is more expressive when the command is intentionally written in a shell language.

JWCCommandSpawn’s shell configuration and escapement model leave room for both approaches.

That is important for a reusable process library. It should not assume that every external tool must be reached through a shell, nor should it prevent callers from using shell semantics when those semantics are the purpose of the command.

Persistent Shell Interaction

A particularly interesting feature of the managed layer is CommandPipe.

Instead of starting a new shell process for every operation, a caller can launch a shell with input and output pipes, then send several commands through the same process.

A typical setup looks like this:

var spawn = new CommandSpawn();

var bash = spawn.GetShell_Bash();
spawn.SetShell(bash);

spawn.Command(
    "bash",
    CommandSpawn.E_PIPE.E_PIPE_STDIN |
    CommandSpawn.E_PIPE.E_PIPE_STDOUT
);

CommandPipe pipe = spawn.CreateCommandPipe();

string first = pipe.Exec("pwd");
string second = pipe.Exec("echo persistent shell");

spawn.Join();

This enables state to persist between commands.

For example:

• the current working directory can change;

• shell variables can remain defined;

• shell functions can be introduced;

• an interpreter can retain loaded state;

• startup cost can be paid once;

• a sequence of related commands can share one environment.

This is fundamentally different from calling Exec repeatedly. Separate process invocations normally begin with separate process state. A persistent pipe creates a continuing session.

Knowing When One Interactive Command Ends

Persistent shells introduce a difficult problem: the shell process remains alive, so reaching end-of-file cannot indicate that one command has completed.

JWCCommandSpawn’s Bash-oriented CommandPipe solves this by wrapping each command with a sentinel.

Conceptually, the command becomes:

(command; echo "***OK*** EXIT:$?")

The managed reader consumes output line by line until it encounters the sentinel. The sentinel identifies the boundary between the current command’s output and the still-running shell session.

It also carries the command’s exit status.

When the parser sees the marker, it:

• extracts the exit code;

• reports a failure for a nonzero result;

• marks the command as complete;

• returns any real output that occurred before the marker;

• stops reading without terminating the shell.

This is a compact but powerful protocol layered over an ordinary shell pipe.

The shell remains a shell. JWCCommandSpawn does not require a custom daemon or external message protocol. Instead, it introduces just enough framing to distinguish one logical request from the next.

Shell-Specific Wrapping and Parsing

The persistent command mechanism is deliberately designed around two delegates:

• one wraps an outgoing command;

• one parses an incoming output line.

For Bash, the wrapper adds the sentinel and exit-code expression. The parser detects the marker and extracts the result.

This architecture recognizes that persistent interaction is shell-specific.

A Python REPL, PowerShell session, database console, or custom command interpreter may need a different framing strategy. The markers, quoting, status checks, and output rules may differ.

By separating command wrapping from output parsing, CommandPipe provides an extension point for future interactive environments.

The general pattern is:

Logical command
      ↓
Environment-specific wrapper
      ↓
Persistent child process
      ↓
Output stream
      ↓
Environment-specific parser
      ↓
Logical command result

That is more flexible than hard-coding all persistent behavior directly into the process class.

Managed Convenience Over a Native Core

The .NET CommandSpawn class is a direct wrapper over the exported native functions.

It holds a native handle and provides managed methods for:

• construction and destruction;

• shell discovery;

• shell selection;

• escapement configuration;

• command execution;

• stream reads and writes;

• pipe closing;

• process joining;

• persistent command-pipe creation.

The wrapper uses JWCEssentials’ managed UTF-8 string structure to transfer text across the native boundary.

This design keeps platform-specific process behavior in native code while allowing higher-level .NET systems to use it without repeatedly writing P/Invoke declarations or operating-system branches.

That is especially useful within the NewAge ecosystem, where managed applications may need to invoke native build tools, shell scripts, generators, media utilities, or supporting processes on both Windows and Linux.

Why a Native Process Layer?

.NET already provides process APIs, so the existence of a native process library is a deliberate architectural choice.

A shared native layer can be valuable when:

• native C++ projects also need the same behavior;

• managed and native systems should follow one process model;

• platform-specific process details are already being handled in native code;

• common exported functions are useful across languages;

• shell and pipe conventions should remain consistent throughout the ecosystem;

• higher-level projects should not each develop separate process wrappers.

JWCCommandSpawn is therefore not simply a replacement for one framework class. It is a common process substrate for a mixed native and managed environment.

A C++ utility and a .NET application can use the same conceptual API, platform behavior, shell definitions, and escapement model.

Process Execution as an Integration Layer

The most important role of JWCCommandSpawn may be as an integration layer.

External tools often represent substantial existing capabilities:

• CMake configures builds;

• compilers produce binaries;

• Git manages repositories;

• FFmpeg processes media;

• Python hosts scripts and machine-learning workloads;

• Bash coordinates Unix tooling;

• PowerShell coordinates Windows administration;

• custom command-line programs expose specialized systems.

A well-designed software ecosystem does not need to reimplement all of those capabilities internally. It needs a dependable way to invoke them, provide input, capture results, and integrate their behavior into a larger workflow.

JWCCommandSpawn provides that bridge.

It allows other projects to treat an external command as a controlled operation rather than an opaque side effect.

Cross-Platform Does Not Mean Shell-Blind

The project’s design again reflects the broader philosophy of the public repository family.

Portability is not achieved by assuming that every platform behaves like one preferred environment. It is achieved by identifying where differences matter and giving those differences a deliberate representation.

For JWCCommandSpawn, the important distinctions include:

• Linux and Windows process creation;

• shell and direct invocation;

• POSIX and Windows quoting;

• Bash, PowerShell, cmd.exe, and Python contexts;

• optional standard-stream connections;

• one-shot and persistent execution;

• process completion and stream completion.

These distinctions are exposed as concepts rather than buried as surprising edge cases.

The result is an abstraction that simplifies process execution without becoming unaware of the systems it abstracts.

A Focused Repository with Broad Reach

Compared with a larger framework, JWCCommandSpawn has a narrow responsibility.

That narrowness is one of its strengths.

Process execution appears in many unrelated domains, but the underlying concerns remain similar. By centralizing them in one reusable library, improvements to escaping, stream behavior, shell support, or platform handling can benefit every project that consumes it.

Potential uses include:

• build orchestration;

• compiler and linker invocation;

• repository automation;

• script execution;

• test harnesses;

• code generation;

• media conversion;

• diagnostic collection;

• command-driven agents;

• interactive development tools;

• external scientific or numerical programs.

JWCCommandSpawn does not need to understand the domain-specific meaning of the command. Its job is to execute that command correctly and make the resulting process manageable.

Its Place Beside CrystalCatalystLibrary

CrystalCatalystLibrary and JWCCommandSpawn address different but complementary forms of operating-system integration.

CrystalCatalystLibrary gives an application a visible and interactive presence:

• windows;

• graphics contexts;

• input;

• pixels;

• cursors;

• clipboard;

• drag-and-drop.

JWCCommandSpawn gives the application access to the external computational environment:

• processes;

• interpreters;

• command shells;

• standard streams;

• exit statuses;

• persistent command sessions.

Together, they cover two major directions of application interaction:

                         Operating system
                         /              \
                        /                \
          Visible application          External tools
          surface                      and processes
                 ↑                      ↑
     CrystalCatalystLibrary      JWCCommandSpawn
                 \                      /
                  \                    /
                   Higher-level software

One creates a place for the application to interact with users. The other creates a controlled path through which the application can interact with other programs.

Its Place in the Public Ecosystem

Within the public JWC software ecosystem, JWCCommandSpawn is the process execution and command-integration layer.

JWCEssentials supplies the common native structures, workspace rules, and staging conventions. JWCCommandSpawn builds on those foundations to launch and communicate with external processes across Linux and Windows. CrystalCatalystLibrary independently uses the same foundation to provide windows, graphics, and native application services.

NewLib can then consume these lower-level capabilities as part of a broader managed and native library collection.

The emerging architecture can be summarized as:

JWCEssentials
    Foundational utilities and workspace conventions
             ↓
    ┌────────┴─────────┐
    ↓                  ↓
CrystalCatalyst    JWCCommandSpawn
Windows, pixels,   Processes, shells,
graphics, input    pipes, external tools
    └────────┬─────────┘
             ↓
           NewLib
Higher-level libraries and integration

JWCCommandSpawn’s purpose can therefore be summarized as:

JWCCommandSpawn provides a portable, shell-aware process layer through which native and managed applications can launch commands, communicate over standard streams, manage process lifetime, and maintain interactive command sessions across Linux and Windows.

Back to overview

View JWCCommandSpawn on GitHub