For example:
% make clean
% make stage
And then configure 3 FLUs:
% echo '{p_srvr, a, machi_flu1_client, "localhost", 39000, []}.' > rel/machi/etc/flu-config/a
% echo '{p_srvr, b, machi_flu1_client, "localhost", 39001, []}.' > rel/machi/etc/flu-config/b
% echo '{p_srvr, c, machi_flu1_client, "localhost", 39002, []}.' > rel/machi/etc/flu-config/c
And then configure a chain to use 2 of those 3 FLUs:
% echo '{chain_def_v1,c1,ap_mode,[{p_srvr,a,machi_flu1_client,"localhost",39000,[]},{p_srvr,b,machi_flu1_client,"localhost",39001,[]}],[],[]}.' > rel/machi/etc/chain-config/c1
... then start Machi e.g.
% ./rel/machi/bin/machi console
... you should see the following console messages scroll by (including a :
=PROGRESS REPORT==== 8-Dec-2015::22:01:44 ===
supervisor: {local,machi_flu_sup}
started: [{pid,<0.145.0>},
{name,a},
{mfargs,
{machi_flu_psup,start_link,
[a,39000,"./data/flu/a",[]]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]
[... and also for the other two FLUs, including a bunch of progress
reports for processes that started underneath that sub-supervisor.]
22:01:44.446 [info] Running FLUs: [a,b,c]
22:01:44.446 [info] Running FLUs at epoch 0: [a,b,c]
22:01:44.532 [warning] The following FLUs are defined but are not also members of a defined chain: [c]
So, the problem is that the chain manager isn't finishing repair
because UPI=[a], and a is a witness, and a can't do the list files etc etc
repair stuff that repairer FLUs need to do.
The best (?) way forward is to add some advance smarts to the
chain manager so that it doesn't propose a UPI of 100% witnesses?
%% So, I'd tried this kind of "if everyone is doing it, then we
%% 'agree' and we can do something different" strategy before,
%% and it didn't work then. Silly me. Distributed systems
%% lesson #823: do not forget the past. In a situation created
%% by PULSE, of all=[a,b,c,d,e], b & d & e were scheduled
%% completely unfairly. So a & c were the only authors ever to
%% suceessfully write a suggested projection to a public store.
%% Oops.
%%
%% So, we're going to keep track in #ch_mgr state for the number
%% of times that this insane judgement has happened.
* Set max length of a chain at -define(MAX_CHAIN_LENGTH, 64).
* Perturb tick sleep time of each manager
* If a chain manager L has zero members in its chain, and then its local
public projection store (authored by some remote author R) has a projection
that contains L, then adopt R's projection and start humming consensus.
* Handle "cross-talk" across projection stores, when chain membership
is changed administratively, e.g. chain was [a,b,c] then changed to merely
[a], but that change only happens on a. Servers b & c continue to use
stale projections and scribble their projection suggestions to a, causing
it to flap.
What's really cool about the flapping handling is that it *works*. I
wasn't thinking about this scenario when designing the flapping logic, but
it's really nifty that this extra scenario causes a to flap and then a's
inner projection remains stable, yay!
* Add complaints when "cross-talk" is observed.
* Fix flapping sleep time throttle.
* Fix bug in the machi_projection_store.erl's bookkeeping of the
max epoch number when flapping.
I'd first thought that having that code there would be a kind of
useful reminder: please move me somewhere else. However, there's
quite a bit there that's "cluster of clusters" stuff and not
appropriate for the current short-term work.
