Skip to content

Externally Scheduled Maintenance Impact on JST Applications#

Note

This page serves as both a business-partner reference and an internal process guide for JST. It only covers schedule maintenance that occurs where JST is not the initiator/driver of the maintenance (i.e. activities initiated by PeopleSoft, DBA's or the Hosting Team). For maintenance/patching/upgrades that JST initiates, please refer to the SLA. Some sections reference JST internal processes and may not be accessible to all readers.

Maintenance Affected Application(s) Cadence Production Timing Testing Notification to JST Team Action
PeopleSoft Refresh AutoSys Monthly Sundays, typically during maintenance window N/A Email from PSSA Nothing
PeopleSoft CPU & CSPU AutoSys Quarterly (CPU), Monthly (CSPU) Sundays, typically during maintenance window Burn in Email from PSSA Usually nothing
PeopleSoft Image AutoSys As needed Weekend, usually multi-day Burn-in in lower environments and/or business UAT testing Email from PSSA Job rescheduling, Stop/Start AutoSys & Agents
PeopleTools Upgrade AutoSys As needed Weekend, usually multi-day Burn-in in lower environments and/or business UAT testing Communication from project coordinator Verification of AutoSys PS Agent & PeopleTools version compatibility, Job rescheduling, Stop/Start AutoSys & Agents
Oracle Hotel Patching AutoSys & ACCE Quarterly Tue-Thu 10am-5pm Burn in for 2 weeks Email from dbnotify@umn.edu Likely restart AutoSys after patching on Thursdays
Linux Kernel Patching AutoSys, Axway, VPSX Monthly Wed 5-7am (non-prd), Sun 5-7am (prd) Burn in for a few weeks None Handled via automation

PeopleSoft Refresh by PSSA#

JST does not take any action for any environment.

PeopleSoft CPU & CSPU by PSSA#

Lower Environments (DEV, TST, QAT, UPG)#

JST does not take any action for lower environments. The expectation is that any regularly scheduled jobs will be delayed or fail if PeopleSoft is unavailable during the CPU/CSPU patching.

Production#

If PSSA plans to implement the PeopleTools CPU in Production during the normal Sunday maintenance window, then no action will be taken by JST. However, if PSSA needs to extend the maintenance window by a few hours in order to accomplish the upgrade, then these are the preferred actions JST will take:

  1. Take all PRD-facing PeopleSoft agents offline (stopped) during maintenance window
  2. Leave the rest of AutoSys and its services up and running
  3. Jobs will queue up in a Pending Machine status until agents are turned back on
  4. Turn agents back on when PSSA gives all clear
  5. Jobs that were paused will begin to process - as will any other jobs that would normally run at the time the agents are turned back on
  6. Run a forecast for the hours that make up the maintenance window extension
  7. Ask customers to put in a ticket if they wish to modify their schedule & attach the forecast to the email

Example

Dear Colleagues,

JST has been informed by PSSA that they need to extend the regular Sunday maintenance window by two hours, going from 6:00 AM - noon to 6:00 AM - 2:00 PM. The extension of the window is for PeopleTools CPU patching.

Attached to this email is a spreadsheet containing a forecast generated in AutoSys Production for the time period 11:59 AM - 1:59 PM on November 12, 2023.

During the extra down time (noon - 2:00 PM) JST will take the AutoSys PeopleSoft Agents offline. This will cause all AutoSys jobs that need to run against PeopleSoft to move to a pausing status of "Pending Machine." AutoSys will then begin to churn through jobs paused in that status when the agents are brought online after a go from PSSA is given to JST.

If your team is OK with things backing up a bit in AutoSys between noon and 2pm and then running once the agents are online (this is often what has been done in years past) - then no action is required by your team.

However, if you need/want to move jobs around or de-schedule them, please submit a TDX ticket to JST (jst-help@umn.edu) by noon on Thursday, November 9th.

Communications For any issues or questions, please send a detailed email to jst-help@umn.edu.

Thank you, Job Scheduling and Transfers Team (JST)

PeopleSoft Image & PeopleTools Upgrades by PSSA#

The PSSA team performs both image upgrades on their PeopleSoft application pillars (i.e. FS Image Upgrade) along with PeopleTools upgrades that JST supports. JST's process for both Image & PeopleTools Upgrades does not differ significantly.

Lower Environments (DEV, UPG)#

No action is taken by JST as jobs in these environments are only run ad-hoc for JST's internal testing.

Lower Environments (TST, QAT)#

JST will unschedule jobs and take agents offline for the duration of this work. For lower environments, jobs are usually paused for 48 hours.

Production#

JST will unschedule jobs and take agents offline for the duration of this work. For production, jobs are usually paused between 11am Saturday and noon Sunday of the upgrade weekend.

Steps JST uses to support Image & PeopleTools Upgrades#

  1. JST looks for project emails and understands overall project schedule at least 2 weeks in advance of TST being upgraded.
  2. JST creates a spreadsheet similar to this sheet to track the steps.
  3. JST's Product Owner sends out an email for each environment outlining the de-scheduling window.
    1. For lower environments, this email should go out 1 week before the anticipated upgrade.
      1. If there is uncertainty on the exact schedule, the first email should outline the overall process (jobs being de-scheduled for 48 hours), and a follow up email will be sent when PSSA picks an exact timeframe for the upgrade.
    2. For the production environment, this email will go out 2 weeks before the upgrade.
  4. For lower environments, PSSA will notify JST at least 24 hours before the upgrade is taking place (if there is any uncertainty on upgrade scheduling) so that JST PO can send a final email, and the JST team can assign a team member to perform necessary actions.
  5. On the day of the upgrade, the on-call team member will be available online 30 minutes before start time to get ready to shut down the respective AutoSys PeopleSoft agents and de-schedule its respective pillar jobs.
  6. JST on-call person will take the AutoSys PeopleSoft agent machine offline (sendevent -E MACH_OFFLINE -n agentname) and then de-schedule jobs using the steps in the method-un-send-events doc.
  7. After jobs are de-scheduled, you can go ahead and stop the respective AutoSys PeopleSoft agents using the AAP template service control.
  8. After the completion of upgrade, JST on-call team member will have to restart AutoSys PeopleSoft agents, then bring the agent machines back online. (sendevent -E MACH_ONLINE -n agentname)
    1. For lower environments this will occur at the end of the 48 hour outage window unless an explicit no-go is received from PSSA.
    2. For the production environment, this will occur Sunday morning after an explicit go is received from PSSA via Slack.
  9. Lastly the jobs can be rescheduled using the reschedule jobs steps.

PeopleTools Upgrade by PSSA#

For PeopleTools upgrades, JST will pay special attention that the version of PeopleTools PSSA is upgrading to is compatible with the version of AutoSys PeopleSoft (PS) Agent Plug-in. If it is not, then that will be discussed with the Project Coordinator as soon as possible for planning purposes. JST may need to upgrade the agent plug-in to a newer version, or coordinate with Broadcom on getting the version supported.

The PSSA team performs PeopleTools upgrades and JST supports during each environment project. This affects JST's application systems such as AutoSys and ps agents. All jobs running on the AutoSys need to be rescheduled before the image upgrades. JST will send a notice to the filetransferandbatch@umn.edu email list asking for jobs to be rescheduled a minimum of two weeks before the production date. These PeopleTools upgrades are usually planned well in advance and a project coordinator will be assigned who will reach out to JST.

Oracle Hotel Database Patching by Go4Enterprise (Oracle DBA's)#

The Go4Enterprise / T3 Database team patches the Oracle hotel databases every quarter. The only JST application that this affects is AutoSys (and secondarily ACCE). As there are three database nodes and only one gets patched at a time, AutoSys should remain functional throughout patching. The only area of concern is that AutoSys doesn't handle re-connecting to nodes that have gone offline, so it is important to make sure that AutoSys has enough connections for each aspect of the application. If it does not, AutoSys will be restarted between 3:45-4pm on Thursday as this is usually a quiet window for jobs. JST will send an email about this, but these emails will be phased out (late 2026) as JST becomes more confident that a rolling AutoSys restart will mitigate business impact.

Linux Patching & Reboot by HEAT#

Each month and as needed, the HEAT team will need all Linux servers rebooted to apply the latest updates and/or kernel. The timing that they initially propose may not be ideal for JST, so it is important to look at that right away. If JST does not like HEAT's timing, JST can instead reboot our servers before their scheduled dates. JST will need to notify HEAT of our plan with a list of servers for them to exclude from their schedule.

Steps JST uses to support HEAT'S Linux Patching & Reboot#

  • For the Axway reboot, JST uses the AAP Scheduled Axway reboot template to reboot the Axway servers before the regularly scheduled window by the hosting team as the servers and services need to be restarted in a specific order.

    • Monthly Schedule

    • DEV - 2nd Tuesday at 2pm

    • TST - 3rd Sunday at 6:05am

    • PRD - 1st Sunday at 6:05am

  • For AutoSys Application Servers, AutoSys Agents & VPSX reboots, JST does not schedule these in advance and instead relies on the hosting team's automation to reboot the servers as all services can resume normally for these applications.

    • Monthly Schedule

    • DEV - Wednesday after the 2nd Tuesday of the month between 5-7am

    • TST - Wednesday after the 3rd Tuesday of the month between 5-7am

    • PRD - 1st Sunday (Production B) of the month between 5-7am