Untitled

# End of shift report - ECS - 22/04/2023 Morning

    Traninee = None

## LHC (machine transitions) 

* ```07:00:00```  PROTON PHYSICS: STABLE BEAMS


## Summary

We were running physics mostly.

    
## Shift flow (time line)

* ```07:00``` Start of shift with a couple of failed attempts to start a stable physics run due to out of sync FLPs.
* ```9:35``` Run 535022 was started which lasted till the beam dump.


## Issues during Shift

1. There were intermittent spikes in the grafana plot during physics runs: [TimeFrame Buffer utilization per EPN](http://alihlt-gw-prod.cern.ch:3325/d/0PsIiR87z/output-rates-and-sizes-3?orgId=1&var-part=All&var-run=All&from=1682140715157&to=1682141615157&refresh=1m&var-partrun=535021%2B-%2B2epwhxXetHF&viewPanel=24) during Physics runs: 
    * For 535021: Made a reply to the existing [bookkeeping entry](https://ali-bookkeeping.cern.ch/?page=log-detail&id=47867) on this issue. 
    * For 535022: [Plot](http://alihlt-gw-prod.cern.ch:3325/d/0PsIiR87z/output-rates-and-sizes-3?orgId=1&var-part=All&var-run=All&from=1682151871425&to=1682162671425&var-partrun=535022%2B-%2B2eq3zhW6jYV&refresh=1m&viewPanel=24))
2. We had  problems with MID
    * For Run535021 :[grafana plot of MID](https://ali-grafana.cern.ch/d/6G6y9TDnk/dataflow?orgId=1&from=1682140502109&to=1682142297109&viewPanel=90). It has a dip. The QC shifter confirmed that the QC plots are fine. Oncall was informed via mattermost direct message.
    * For Run535022 :[Spike structures in the grafana plot of MID](https://ali-grafana.cern.ch/d/6G6y9TDnk/dataflow?orgId=1&from=1682154658074&to=1682158091665&viewPanel=90) 
3. We had wiggles in the [grafana plot of HMPID](https://ali-grafana.cern.ch/d/6G6y9TDnk/dataflow?orgId=1&from=1682140807326&to=1682142602326&viewPanel=74) during 535021. 
    * Oncall was informed via mattermost direct message.
4. During 535021 its-stf-decoder crashed during "PHYSICS" on several epns - ITS. 
    * ***```Stopped the run```, made a [bookkeeping entry](https://ali-bookkeeping.cern.ch/?page=log-detail&id=47880) and started a new run***.



## From the previous shifter
None

## To the next shifter
1. The issues mentioned above with MID and HMPID are informed to the respective oncalls via mattermost. MID oncall responded that it needs investigation. HMPID oncall is yet to respond to the message. 
 
 ## To the RM/RC
 None




****
%bogus%
%Rec Error in%
%Raw Error%
%We are trying to send an oldest possible timeslice%
%NOT retrieved!!!%
%An error occurred during retrieval%
%task terminated with error: numactl%
%We could NOT retrieve the object%
%Orbit%
%Requested resource does not exist:%