Small items not worth separate talks.

1. Minor Technical Issues
Thread Initialization
Continuation Tracking
Time and std::chrono

2. Thread Initialization

3. Current thread architecture
 Event threads and dedicated threads
 Event threads dispatch events from queues and do socket I/O.
 Dedicated threads execute a single event.
 Threads are kept in a global table
 Each event thread has thread local storage which contains an Ethread
instance.

4. Thread Initialization
 Thread initialization is done in a very unsafe manner.
 Threads are started.
 “Later”, but not too much later, the main thread updates data structures in the
other threads.
 Issues
 Obviously unsafe, despite it seeming to work.
 Thread initialization is monolithic due to having to put all the logic in one
intervention call site to avoid more race conditions.
 Event loop update (TS-4260) removed an unused method that was used by another
patch (NUMA), need to add it back.

5. TS-4265
Update thread initialization
 Change thread arrays to be arrays of structures to consolidate thread
information.
 Do thread initialization via continuations.
 Together these enable different parts of the process start logic to execute
code in another thread, during thread startup, without races with each other
or the thread.
 Intended for event threads.
 Not really needed for dedicated threads.

6. Start events
 Implementation – re-use the “oneevent” member for event threads.
 Currently only used by dedicated threads.
 Change event threads to execute this event if set.
 Put a core event there that then executes a queue of other events.
 Thread initialization means putting an event in this “spawn queue”.

7. Scheduling at spawn
 EThread::schedule_spawn
 Single continuation per thread that is called first for regular thread.
 EventProcessor::schedule_spawn
 Multiple functions that are invoked when an event thread is started.
 Built on top of thread schedule spawn.

8. Other changes
 Event processor startup
 Net processor startup
 NUMA and processor affinity

9. Continuation Tracking

10. Track original source of continuations
 Originally designed for plugin priorities.
 Need to know which plugin for a continuation to get the priority correct.
 Allows access to plugin registration and related data.
 Requested as an independent feature.
 For debugging to track continuation problems back to specific plugins.
 Potentially useful for disabling plugins.

11. Implementation
 Borrow a technique from per client IP debugging.
 Add per thread data item which points at the plugin registration data.
 Track current “plugin context”
 During plugin load any continuations created are tagged with the plugin data.
 During event dispatch
 Current plugin context is saved.
 Context is loaded from continuation.
 After event the old context is restored.
 When a continuation is created it gets the current context.

12. Consequences
 Plugin registration data pointer becomes continuation context.
 Need “core” plugin to represent continuations from the core and not a plugin.
 Can go off track if plugins “trade” continuations.
 Dispatch can check plugin state before calling continuation handler.

13. Todo
 Switch to C++ eleventy thread storage.

14. IP Allow Extension

15. IP Allow
 Quinn’s project.
 Extend IP allow rules to control outbound connections based on the origin IP
address.
 Enable control of IP Allow rule enforcement in remap rules.
 Currently if full DENY then connection closed before remap.

16. IP Allow implementation
 Add “dest_ip” key to ip_allow.config.
 Add “ip_allow” as a remap filter so it can be turned off and on.
 On by default
 Disable able early / accept check if any remap rule has IP Allow disabled.
 Global effect –if any remap rule bypasses IP Allow then the fast check is not done.

17. Example – allow outbound only to the
10/8 net
dest_ip 10.0.0.0-10.255.255.255 action=ALLOW
dest_ip 0.0.0.0-255.255.255.255 action=DENY

18. Example - remap
# ip_allow.config
src_ip=127.0.0.1 action=ip_allow method=ALL
src_ip=::1 action=ip_allow method=ALL
src_ip=0.0.0.0-255.255.255.255 action=ip_deny method=PUSH|PURGE|DELETE
src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny
method=PUSH|PURGE|DELETE
# remap.config – disable ip_allow for the remaps, let everything else be denied
.deactivatefilter ip_allow
remap ….allow purge….
remap … allow purge….
.activatefilter ip_allow

19. std::chrono
Time keeping in C++ eleventy

20. std::chrono and time
 std::chrono is a library for handling time intervals.
 NOT calendar time! That’s a very different thing.
 Instances are efficient
 Only the time unit is stored.
 Other properties are class properties known by the compiler.
 Conversions are known at compile time and inlined.
 Conversions are semi-automatic.
 Conversions to finer resolution are automatic.
 Conversions to coarser resolution require explicit conversion.
 Information is never unintentionally lost.

21. std::chrono duration and time points
 A duration is a length of time.
 A timepoint is a specific point in time.
 Represented internally as a duration and an epoch.
 Epoch is a globally defined and known point in time.
 E.g. standard UNIX epoch 1 Jan 1970.

22. std::chrono conversions and arguments
 Expressing exactly what time unit is desired is easy.
 std::chrono::milliseconds(5)
 std::chrono::hours(1)
 Automatic conversions let these be passed to any parameter that is at least as
fined grained.
 If methods take std::chrono::nanoseconds then passing time is easy.
 Caller does not need to know the parameter granularity unless it is coarser in which case
it is good to get a compile time error.
void leif_nap(std::chrono::nanoseconds length);
// ...
{
leif_nap(std::chrono::minutes(30));
leif_nap(std::chrono::days(2));
}

23. ATS example
// from P_CacheInternal.h
int cache_config_mutex_retry_delay = 2;
trigger = mutex->thread_holding->schedule_in_local(this,
HRTIME_MSECONDS(cache_config_mutex_retry_delay), event);
// vs
ts::milliseconds cache_config_mutex_retry_delay{2};
trigger = mutex->thread_holding->schedule_in_local(this,
cache_config_mutex_retry_delay, event);

24. std::chrono and clocks
 std::chrono supports clocks
 system_clock
 monotonic_clock
 high_precision_clock
 My research indicates that in real life system_clock and high_precision_clock are
the same thing and sometimes monotonic_clock as well.
 Time points are based on a specific clock because that clock embodies the
epoch.
 Durations are clock independent.

25. Active Projects
 TS-974: Partial Object Caching
 TS-4260: Event loop improvements
 TS-4999: Plugin priority
 TS-4532: std::chrono
 TS-3347: make ink_assert a no-op in release mode.
 TS-4593: Extend IP allow to outbound connections.