Truth in the Flip: Building the Farm Before the Harvest

The current Quant_IDQE.tkr run has reached another useful horizon:

7774800000000 flips → heads: 50+1.2e-04% (Z:+0.7446)
| a: 50+1.1e-04% (Z:+0.6663)
| aHeads: 50-1.2e-05%
| aTails: 50+2.2e-04%
| aSame: 50-5.1e-05%
| aDiff: 50+2.6e-04%
| wallclock: 24.06:27:36
| fps: 3707835
| Z(1.96) in 2.09:20:31
| Z(3.00) in 6.00:23:57

That is 7.7748 trillion flips, after more than twenty-four days of computation, with the tracker still advancing at roughly 3.7 million flips per second.

The current Z-scores are not a conclusion. They are a snapshot.

That distinction matters.

One of the recurring ideas behind Truth in the Flip has been that a long-running stochastic process should be allowed to remain a process. A momentary excursion is interesting, but it is not the same thing as a settled result. A statistic can move toward significance, away from it, through it, and back again.

So the tracker continues.

But while the tracker has been running, something else has grown around it.

The analysis machinery has become a project of its own.

Building the Farm

The command-line analysis system is called TruthInTheFlip Farm.

The name started naturally enough: tracker data comes in, processes work over it, statistics are harvested from it.

But the architecture has become more interesting than a collection of reports.

The Farm is built around composition.

At a high level, a command describes three separate concerns:

source → process → output

A source identifies tracker records and applies selection or transformation.

A process determines how those records are interpreted.

An output layer projects the resulting metrics.

A simple example might look like:

csv tracker file Quant_IDQE.tkr Total ZScore AnticipatedPercentage

Conceptually, that is closer to:

csv(
    tracker(
        file("Quant_IDQE.tkr")),
    Total,
    ZScore,
    AnticipatedPercentage)

than it is to the usual command-line model of switching over a bag of unrelated options.

The command grammar is typed.

A method returning a tracker selector can be supplied anywhere another operation expects a tracker selector. A segmentation operation returns a process whose products have their own metric catalog. Another operation can consume those products.

That compositional structure has become the central idea of the Farm.

From Trackers to Segments

Individual tracker records are useful, but a long stochastic run is often more interesting when viewed as a sequence of intervals.

The Farm therefore supports segmentation.

For example:

segment

turns a tracker population into SegmentStats.

A segment can describe things such as:

its beginning and ending tracker records,
its best True Z excursion,
its ending True Z,
its mean True Z,
anticipation behavior,
underlying heads behavior,
and the fraction of its path spent above or below useful thresholds.

Then there is another level:

segment_agg

which operates over populations of SegmentStats and produces a SegmentAggregate.

Conceptually:

Tracker
    ↓
segment
    ↓
SegmentStats
    ↓
segment_agg
    ↓
SegmentAggregate

That hierarchy turned out to matter enormously once metric functions became composable.

A Small Metric Expression Language

Originally, CSV projection mostly meant selecting a metric name:

ZScore

or walking through a nested property:

End.AnticipatedPercentage

The . already had a clear meaning:

.    metric/property traversal

Then metric functions arrived.

The syntax uses #:

abs#EndTrueZ

or:

mean#anticipatedTails

So the basic vocabulary became:

.    metric/property traversal
#    metric function application

The next question was unavoidable.

What happens when a function needs more than one argument?

That gave us:

,    function argument separation

and suddenly expressions such as these became possible:

lerp#MinZHeads,MaxZHeads,.5

pearson#ZScoreHeads,ZScoreTails

pearson#ZScoreHeads,abs#ZScoreTails

The surprising part is that the grammar does not need parentheses.

Arity Supplies the Structure

Consider:

clamp#value,min,max

The reflected clamp method tells the parser that it takes three parameters.

That fact supplies the structure.

The parser does not need this:

clamp(value,min,max)

because the method signature already says how many expressions must be consumed.

This becomes especially interesting with nesting.

One of the stress tests used while finishing the parser was:

clamp#mean#abs#stddev_sample#AnticipatedPercentage,-1,0

And then, because trusting a parser too early is dangerous, I tried:

clamp#clamp#mean#abs#stddev_sample#AnticipatedPercentage,-1,0,-1,-.5

It worked.

The structure can be understood roughly as:

clamp
    clamp
        mean
            abs
                stddev_sample
                    AnticipatedPercentage
        -1
        0
    -1
    -.5

No parenthesis stack is needed in the text representation.

The recursive parser knows when an expression ends because the reflected method arity tells each call how many parameters it owns.

That turned out to be one of my favorite properties of the design.

Scalar and Aggregate Parameters

There is another distinction hiding inside the method signatures.

A scalar metric parameter such as:

double value

means:

evaluate this expression at the current process level.

An aggregate parameter such as:

List<double> values

means:

sample this expression from the child process population.

That gives metric functions process semantics without requiring each function to know anything about the Farm.

A scalar function such as:

abs#EndTrueZ

stays where it is.

An aggregate function such as:

mean#ZScore

moves one level into the process below it, collects the values, and then evaluates the function.

This means nested aggregates naturally descend through the process graph.

For example:

mean#mean#ZScore

can mean:

collect ZScore over tracker records,
calculate a mean for each segment,
then collect those means across an aggregate of segments,
and calculate another mean.

The syntax and the process hierarchy cooperate.

That was not originally designed as a grand expression language. It emerged from asking what the type signatures already knew.

Statistical Functions

Once the plumbing was capable of carrying functions properly, it made sense to give the Farm a useful statistical vocabulary.

The scalar side now includes operations such as:

abs
negate
square
sqrt
ln
pow
offset
offset50
scale
ratio
clamp
lerp

The aggregate side includes descriptive statistics such as:

count
sum
mean
min
max
median
variance_population
variance_sample
stddev_population
stddev_sample
rms
mean_abs
covariance_population
covariance_sample
pearson

I deliberately prefer explicit names such as:

stddev_sample

and:

stddev_population

instead of hiding the n versus n - 1 choice behind an ambiguous stddev.

The Farm is supposed to make analysis easier, not make statistical assumptions invisible.

Pearson as a Parser Test

Pearson correlation became one of the most useful tests of the new machinery.

This expression:

pearson#ZScoreHeads,ZScoreTails

requires two independent aggregate populations.

That exposed an important implementation issue: initially both parameters were accidentally drawing from the same stored series.

The symptom was wonderfully obvious.

Every Pearson coefficient was 1.

Of course it was.

The Farm had effectively been calculating:

pearson(x, x)

instead of:

pearson(x, y)

Once aggregate state became parameter-specific, the result changed immediately.

A real run produced:

mean#anticipatedTails,"pearson#ZScoreHeads,ZScoreTails"
12525048877.594,-1.000000000000002
25000093183.95,-0.9999999999999996
24999860225.76,-1.0000000000000002
24999952853.39,-1
25000013324.862,-1.0000000000000007
25000014662.32,-0.9999999999999997
24999941909.36,-1.0000000000000002

The tiny excursions outside -1 are ordinary floating-point roundoff.

The interesting part is that ZScoreHeads and ZScoreTails are behaving as essentially perfect opposites over those sampled segment populations.

That is not necessarily mysterious. Those quantities are closely related mathematically.

But as an engineering test, the result was excellent.

The Farm had successfully:

parsed a two-argument aggregate function,
sampled two separate child-process expressions,
kept their observations aligned,
invoked the reflected statistical function,
and projected the resulting expression as CSV.

The Expression Survives the Pipeline

There was another design decision that paid off during this work.

A metric expression should remain its own canonical name.

So this:

pearson#ZScoreHeads,ZScoreTails

should not become:

pearson_ZScoreHeads_ZScoreTails

just because CSV uses commas structurally.

Instead, the CSV writer does what CSV is supposed to do:

"pearson#ZScoreHeads,ZScoreTails"

The expression remains intact.

That matters downstream.

I tested a Pearson expression through the plotting path as well, and the formula appeared correctly in the chart.

That means the same expression can travel through:

Farm
    ↓
CSV
    ↓
Pandas
    ↓
plot

without losing its identity.

The column name is not merely an implementation artifact.

It is a compact description of what was calculated.

A Human-Readable Segment Report

Not every analysis should require CSV or Python.

The Farm also contains:

segment_report

which is a curated human-readable report over segmented tracker data.

It supports progressive report grades and can expose increasingly detailed material such as:

segment count,
Edge Excursion Score,
Edge Settlement Score,
Edge Persistence Index,
anticipation geometry,
underlying heads geometry,
threshold frequencies,
detailed per-segment rows,
Pearson correlation diagnostics,
retained anticipation,
settlement-adjusted anticipation,
standout segments,
and total file compute time.

Three summary quantities have become especially useful:

Edge Excursion Score
    median(best TrueZ per segment)

Edge Settlement Score
    mean(end TrueZ per segment)

Edge Persistence Index
    settlement × fraction positive

These do not replace the underlying tracker.

They provide different views of how an apparent edge behaves over time.

An excursion can be large and still fail to settle.

A positive ending average can still be supported by very few segments.

Persistence asks yet another question.

This is exactly why I wanted the Farm to preserve multiple perspectives rather than collapse a long run into one headline number.

The Farm and the Tracker

There is something satisfying about the timing of all this.

The Quant_IDQE.tkr run was not paused while the Farm was being built.

It kept moving.

At the snapshot that opened this post:

7,774,800,000,000 flips

the tracker reported:

heads Z: +0.7446
anticipation Z: +0.6663

with anticipation components including:

aHeads: 50-1.2e-05%
aTails: 50+2.2e-04%
aSame:  50-5.1e-05%
aDiff:  50+2.6e-04%

Those values are worth recording because they describe this point in the run.

They should not be mistaken for its final state.

The tracker has already demonstrated that long stochastic runs can wander substantially across different horizons.

That is part of the experiment.

Waiting Without Being Idle

When this run began, the main question was about the tracker.

By the time it ends, the analytical environment waiting for it will be considerably stronger than the one that existed at the start.

The waiting period produced:

a reusable Farm process layer,
compositional tracker selectors,
segmentation,
second-level segment aggregation,
a human-readable segment report,
reflected metric catalogs,
recursive metric paths,
scalar metric functions,
aggregate statistical functions,
multi-parameter functions,
numeric literals,
nested structural arity,
Pearson correlation,
canonical CSV expression names,
and a path into Pandas and plotting.

That changes what can be asked of the final tracker.

Instead of one report containing the questions I happened to anticipate beforehand, the tracker can now be interrogated compositionally.

A new metric expression can be written at the command line.

A population can be segmented differently.

A statistic can be applied at one process level or another.

A result can become CSV without adding a custom report.

And the same canonical expression can survive into downstream analysis.

Before the Harvest

The name Farm now seems more appropriate than when I first used it.

A farm is not the harvest.

It is the infrastructure that makes many harvests possible.

The long-running tracker is still out there accumulating evidence.

At this moment, it has crossed 7.77 trillion flips and is still running.

There is no need to force an ending because a statistic happens to look interesting today.

When Quant_IDQE.tkr finally reaches its intended horizon, I will preserve the completed tracker and examine it from several directions.

The Farm will be ready.

And perhaps that is the most useful development from this waiting period.

We did not merely wait for a number.

We built better ways to ask what the number means.

Documentation

The Farm has several layers of documentation:

github.com/johnwaynecornell/TruthInTheFlip