Joseph Tracy. Infrastructure Engineer, Storage and Automation.

About Joseph Tracy

I am an IT infrastructure engineer in Seattle with over two decades in systems and network administration and systems engineering. My work has covered Linux and Windows server environments, enterprise scale-out storage, virtualization, campus networking, and the identity, collaboration, and monitoring systems that run on top of them. The LinkedIn profile carries the same history in resume form.

Over that time I have worked as sole IT administrator for an entire organization, network operations escalation point for more than 750 servers across global data centers, campus infrastructure administrator, data center operations analyst, and, since 2018, senior backline support engineer for enterprise storage.

What I do now

I am a senior support engineer on the Dell PowerScale team at Dell Technologies. PowerScale, known as Isilon before 2020, is a scale-out network-attached storage (NAS) platform. A PowerScale cluster is a group of storage nodes running the OneFS operating system, which presents one file system and one namespace across every node. Clients reach it through file protocols (NFS and SMB), the S3 object protocol, and HDFS for Hadoop workloads. Scale-out means capacity and performance grow by adding nodes to the cluster rather than replacing the system.

I work at the backline level, supporting enterprise customers directly and the frontline engineers who escalate cases to me. Most cases reach me after the earlier support tiers have ruled out the common causes, so the work usually requires reading the system's logs, configuration, and state in detail. Others arrive as management escalations, or as collaboration requests from frontline engineers who need help on a case. I also join customer-facing calls, and when the team needs it I take frontline cases and work them directly. My role is Senior Enterprise Distributed Network Attached Storage (NAS/SAN) Technical Support Services Engineer. My product scope is PowerScale and OneFS.

Most of my casework falls in one area of the platform: the services that connect a cluster to Dell's support infrastructure and report on the cluster's own health. Dell Technologies Connectivity Services (DTCS), called SupportAssist before OneFS 9.10, carries the cluster's connection to Dell support, and the telemetry it sends reaches Dell AIOps, formerly CloudIQ. That includes break/fix work on:

  • remote support connectivity through DTCS, including migrations from legacy Secure Remote Services to SupportAssist and DTCS, and the failures that block them
  • telemetry transmission from the cluster to Dell AIOps
  • the cluster health check framework, when its evaluations stop running or stop producing accurate results
  • the cluster event log and its notification paths, including SMTP email alerting, quota notifications, and event delivery
  • the OneFS job engine, including failed maintenance jobs that affect data protection and file system integrity, such as FlexProtect, AutoBalance, Collect, and MultiScan
  • certificate and management-plane failures across these services

I also take cases involving the OneFS API, an area I am working to know in more depth.

Alongside casework, I lead technical investigations during production-impacting incidents, such as node-down events and failed upgrades, and work with engineering and field teams on complex product issues, including defect reproduction.

I write knowledge base articles and process documentation for my area: how to research these cases, how to run the engagement, and how to build the action plan for diagnosing, mitigating, and resolving them. I mentor colleagues on troubleshooting method and on handling escalations.

Escalation management is a significant part of the role. I act as the technical escalation and collaboration point for high-complexity, politically sensitive service requests, working with colleagues, account and incident managers, and technical support leaders, and I coordinate cross-functional technical strategy and de-escalation for critical and elevated service requests. I work complex, multifaceted problems through evidence-based analysis and collaboration across disciplines.

How I build automation

I have used scripting to remove repetitive work throughout my career. At Horizon House I wrote PowerShell that created more than 70 Active Directory accounts from HR spreadsheet data, with validation checks built in. At Concord Technologies I deployed registry patch updates to 215 servers across three data centers in a single PowerShell run. In my current role I use Bash extensively, since OneFS is built on FreeBSD, including to work through log volumes too large to read by hand.

The projects and articles on this site are where my current automation work is published.

Technical areas

  • Storage: Dell PowerScale and OneFS (formerly Isilon), scale-out NAS, NFS and SMB file services, cluster health, event notification, the job engine, Dell Technologies Connectivity Services, and telemetry to Dell AIOps
  • Operating systems: Linux (Red Hat Enterprise Linux, Ubuntu, CentOS) and Windows Server, including Active Directory, DNS, DHCP, and Group Policy
  • Virtualization: VMware vSphere and Hyper-V
  • Networking: campus switching, wireless infrastructure, VLAN design, and TCP/IP troubleshooting
  • Monitoring and service management: SolarWinds Orion, Microsoft System Center Operations Manager (SCOM), PagerDuty, Datadog, Remedy, and Jira
  • Scripting and automation: Bash, PowerShell, and Python
  • Collaboration and messaging: Exchange and Office 365 administration, including migration from on-premises Exchange to Office 365
  • Support practice: escalation management, technical investigations during production incidents, defect reproduction with engineering, evidence-based analysis, follow-the-sun handoff, knowledge base and process documentation, and mentoring

Current focus and direction

My established experience is in enterprise storage and the systems around it: Dell PowerScale and OneFS, Linux and Windows Server administration, Active Directory, virtualization, campus networking, and monitoring.

I am moving my career toward DevOps, automation engineering, and systems automation, through coursework (listed under Training) and hands-on work in my own lab. That work is newer than my operations experience and is not yet production experience.

Work history

  • Senior Enterprise Distributed Network Attached Storage (NAS/SAN) Technical Support Services Engineer

    Dell Technologies December 2018 to present

    Backline break/fix support for Dell PowerScale (formerly Isilon) and the OneFS operating system, for enterprise customers and the frontline engineers who escalate to me. Technical escalation and collaboration point for high-complexity, politically sensitive service requests, working with account managers, incident managers, and support leaders, and coordinating cross-functional technical strategy and de-escalation for critical and elevated cases. I mentor colleagues on troubleshooting method and escalation management, and write knowledge base articles and process documentation for connectivity and telemetry cases.

    Contributions: I redesigned documentation workflows and handoff practice for follow-the-sun coverage, and contributed as part of a cross-functional team to a Salesforce-to-Slack case engagement method that moved from alpha to beta and is now used across global PowerScale technical support. I also tested and gave detailed feedback on internal generative AI tools for support workflows.

  • System Analyst II, Operations Command Center

    TEKsystems, on site at Starbucks September to November 2018

    Short-term contract covering a data center operations analyst on extended leave. Monitored business-critical systems, and provided first and second level support across Red Hat Enterprise Linux, Ubuntu, CentOS, Windows Server, and IBM i on AS/400. Administered VMware vSphere, including guest migrations and resource rebalancing. Monitored data center environmental systems, managed data center access for partners and vendors, processed change requests for remote sites, racked equipment, and ran PowerShell for routine automation. Tools included BMC Control-M, Remedy, Jira, Confluence, PagerDuty, Datadog, and Slack.

  • Network Systems Administrator III

    Horizon House June 2016 to February 2018

    Campus infrastructure for a continuing care retirement community: 196 wireless access points, more than 55 virtualized Windows servers, and the network switching beneath them. Escalation point for the help desk. Designed the VLAN configuration for a campus-wide IP security camera deployment, monitored the network and servers with SolarWinds Orion and SCOM, and migrated the Orion and Web Help Desk databases from SQL Server 2012 to 2014. Deployed a Windows Server 2012 print server, Group Policies, and security groups, and managed DHCP scopes. Wrote PowerShell to create more than 70 Active Directory accounts from HR data with built-in validation, to report Active Directory discrepancies to HR, and to detect and manage Windows service failures that would otherwise have taken the campus door card readers offline.

  • Senior Network Operations Center Systems Administrator

    Concord Technologies May 2015 to June 2016

    Escalation point for premium support and management across more than 750 physical and virtualized Windows and Linux servers in global data centers, including a remote site in Europe. Managed monitoring tools and ticket queues, tuned alert thresholds to reduce false positives, analyzed and documented alert notifications, and standardized procedures in the enterprise ticketing system for recurring incidents. Used PowerShell to create remote shares, automate backups of billing archive data, and deploy registry patch updates to 215 servers across three data centers in a single run, and worked with the development team to improve service logging and automate recovery for virtualized Windows servers. Shared bi-weekly on-call rotation.

  • Information Technology Administrator

    Seattle Children's Theatre December 2012 to May 2015

    Sole IT administrator for a nonprofit theatre, responsible for 50 Linux thin clients, 25 Windows workstations, 14 Windows Server 2008 R2 servers, Active Directory, DHCP, DNS, and Exchange. Led the migration from on-premises Exchange 2010 to Office 365, added a Windows Server 2008 terminal server, managed Backup Exec backups, and scripted multi-tier backups for the SQL Server databases. Advised management on IT budgeting and on the practical effect of funding decisions on infrastructure. Provided around-the-clock on-call support as the organization's only technical contact.

  • Technical Support Analyst and Junior Network Administrator

    McKesson April 2008 to April 2010

    Support for the Practice Partner electronic medical records and billing software, for practices ranging from single clinics to multi-practice enterprises, under HIPAA. Resolved data integrity issues in Oracle, SQL Server, and FairCom c-tree databases, evaluated customer hardware and network infrastructure, and served as technical lead for remote multi-clinic go-live deployments. Helped facilitate two annual Practice Partner user group meetings and training classes. Recognized as a Technical Performance Superstar.

  • Customer Support Representative, then Air Fulfillment Operations Agent

    Expedia January 2005 to April 2008

    Handled in-travel and post-travel changes and elevated customer service issues for customers, vendors, and travel agents, and coached new hires moving from the training room to the call center as a training advocate. From May 2007, as an air fulfillment operations agent, worked with vendors, suppliers, and customers to resolve system and process failures in the Sabre and Worldspan air booking systems, supported the fulfillment team's continuous improvement work, and provided crisis management support during travel disruptions.

  • Senior Software Tier 2 Support, then Research Team Tier 3 Support

    Stream International 2003 to 2004

    Tier 2 escalation contact for the support floor, then tier 3 research: reproducing reported defects in the application, working with developers on defects and feature gaps, and documenting resolutions in the knowledge base. The tier 3 role required passing a 140-hour income tax course with a score of 90 percent or higher before taking support calls on the software.

Training

The entries below are course completions, not certifications.

Enterprise storage

  • PowerScale Administration, 40 hours, and PowerScale Advanced Administration, 40 hours, both instructor-led through Dell Technologies.
  • PowerScale advanced Bash scripting for OneFS, and PowerScale S3 fundamentals.
  • OneFS troubleshooting methodology and knowledge transfer coursework.

Linux and Red Hat

  • Red Hat System Administration I and II (RH124 and RH134), instructor-led, on RHEL 8 and 9, with System Administration I repeated for RHEL 10, and the RH024 technical overview.
  • Red Hat Certified System Administrator v8 preparation, an 80-hour, two-week course through Dell Technologies. This is a course completion; I have not taken the RHCSA exam.
  • Red Hat Ansible Basics: Automation Technical Overview (DO007) v2.0.
  • Introduction to Linux, 24 hours, through Dell Technologies.
  • UNIX for network administration, and UNIX operating system coursework mapped to CompTIA Linux+, both at Seattle Central College.

Problem-solving method

  • Kepner-Tregoe structured problem solving, 45 hours over one week, through Dell Technologies.
  • Learning How to Learn, University of California via Coursera.

Automation and scripting

  • Getting Started with PowerShell 3.0 and Advanced Tools and Scripting with PowerShell 3.0, Microsoft Virtual Academy.
  • Microsoft PowerShell concepts and features through Dell Technologies, and reporting with PowerShell HTML through Pluralsight.
  • Introduction to programming with Python, Microsoft Virtual Academy.
  • Programming concepts and fundamentals in C#, Seattle Central College.
  • Git version control fundamentals, and GitHub for beginners.

Containers and cloud

  • KubeAcademy Containers 101 and Kubernetes 101, through Dell Technologies.
  • AWS Skills Centers: Demystifying Generative AI, Amazon.

Networking and Windows infrastructure

  • Cisco networking sequence at Seattle Central College: network fundamentals, routing protocols and concepts, LAN switching and wireless, and WAN and VoIP.
  • Network essentials mapped to CompTIA Network+, and computer hardware mapped to CompTIA A+.
  • Windows Server 2008 administration, Active Directory configuration, and network infrastructure and TCP/IP configuration.
  • Windows 7 configuration, SharePoint and Exchange enterprise applications, and introduction to network security.

Monitoring and security

  • Installing System Center 2012 Operations Manager, Pluralsight.
  • Building your cyber security vocabulary, Pluralsight.

Generative AI

  • Generative AI Foundations and Generative Artificial Intelligence Concepts, through Dell Technologies, covering ethics, model concepts, and the AI ecosystem.
  • Boosting productivity with AI tools, Dell Technologies.
  • Hands-on evaluation of internal generative AI tools for support workflows, described in the Dell entry under Work history.

Web and front end

  • Deep JavaScript Foundations and Introduction to JavaScript Programming, both taught by Kyle Simpson through Frontend Masters.
  • Hand-coded XHTML and CSS at Seattle Central College, HTML and CSS through Codecademy, and workshops in HTML, CSS, and JavaScript through Galvanize.

Education

  • Seattle Central College, Network Design and Administration, 2010 to June 2012 (117 credits).
  • Centennial Job Corps Center, Vocational Certificate, Computer Services, 2001 to 2003.

Lab and boundaries

Everything published on this site is built and tested on my own time, on a personal virtualization host running Linux and storage virtual machines. I use it to test scripts against realistic file systems and services before publishing them, and to rehearse failure scenarios under controlled conditions.

The lab is a personal machine. It is separate from production environments and from employer hardware, and I make no claim to running infrastructure at scale outside my employment. Nothing on this site is derived from customer environments or internal systems; it is built from public documentation and my own lab work.

I am open to contract and consulting work in infrastructure automation, diagnostics, storage operations, and modernization reviews, and to full-time roles where escalation-grade troubleshooting and automation ownership are the substance of the job and not a line in the description. If any of that overlaps with what you are building, the connect page reaches me directly.