================================================================================
              udev — Device Manager Reference
================================================================================
Author      : Tushar Sonawane
Date        : 09 September 2026
============================================================

TABLE OF CONTENTS
================================================================================

  1.  udev — Overview
  2.  Verifying installation
  3.  Boot Integration — rc.udev
  4.  udevadm — The Primary Tool
  5.  Writing udev Rules
  6.  Disk Renaming and Persistent Symlinks
  7.  Permissions, Ownership, Groups
  8.  Hardware Event Monitoring
  9.  Hardware Database (hwdb)
  10. Advanced Patterns
  11. Debugging
  12. Quick Reference Card
  13. Conclusion
================================================================================
1.  udev — OVERVIEW
================================================================================

udev is the Linux userspace device manager responsible for dynamically managing
hardware devices and maintaining the /dev device tree. It listens for kernel
uevents, creates and removes device nodes, applies permissions, generates
persistent device names and symlinks, and can execute custom actions when
hardware is added, removed, or changed.

Key responsibilities:
  - Detects hardware as it appears or disappears.
  - Creates and manages device nodes under /dev.
  - Applies device permissions, ownership, and groups.
  - Creates persistent device symlinks (by-id, by-uuid, by-label, etc.).
  - Executes custom rules and scripts based on device events.
  - Maintains a runtime device database for fast device lookups.

udev operates through the udevd daemon, which receives events from the kernel
and processes them according to rules stored in system and administrator-defined
rule files.

--------------------------------------------------------------------------------
1.1  Architecture 
--------------------------------------------------------------------------------

  +----------------------------------------------------------+
  |  Hardware (disk, USB, NIC, GPU...)                       |
  +---------------------------+------------------------------+
                              | kernel uevent (netlink socket)
  +---------------------------v------------------------------+
  |  /sbin/udevd  (eudev daemon)                            |
  |  -> reads /lib/udev/rules.d/    (system rules)          |
  |  -> reads /etc/udev/rules.d/    (your custom rules)     |
  |  -> matches SUBSYSTEM, KERNEL, ATTR, ENV...             |
  |  -> creates /dev symlinks, sets permissions, runs RUN   |
  +---------------------------+------------------------------+
                              | started/stopped by:
  +---------------------------v------------------------------+
  |  /etc/rc.d/rc.udev   ( BSD init script)        |
  |  called from /etc/rc.d/rc.S  at boot                    |
  +----------------------------------------------------------+

================================================================================
2.  INSTALLATION VERIFICATION.
================================================================================
  # Check eudev version:
  udevadm --version (3.2.11)

  ls /var/log/packages/ | grep eudev
  # eudev-3.2.11-x86_64-1

  which udevadm
  # /sbin/udevadm

  ls /sbin/udevd
  # /sbin/udevd

--------------------------------------------------------------------------------
2.2  Verify eudev is Working
--------------------------------------------------------------------------------

  # Check daemon is running
  ps aux | grep udevd
  # /sbin/udevd --daemon

  # Check udev database is populated
  ls /run/udev/data/ | wc -l

  # List block devices recognized by eudev
  udevadm info --query=all --name=/dev/sda

================================================================================
3.  BOOT INTEGRATION — rc.udev
================================================================================

# udevd is started and stopped by the BSD init script:
/etc/rc.d/rc.udev

# Make sure script is executable.

--------------------------------------------------------------------------------
3.1  How rc.udev Works
--------------------------------------------------------------------------------

The script (/etc/rc.d/rc.udev) does the following at boot:

  1. Sanity checks (tmpfs support)
  2. Mounts devtmpfs on /dev if not already mounted
  3. Mounts devpts under /dev/pts
  4. Starts /sbin/udevd --daemon
  5. Runs: udevadm trigger --action=add   (replays add events for existing hw)
  6. Runs: udevadm settle                 (waits for queue to drain)

At shutdown (/etc/rc.d/rc.0 or rc.6):
  7. Kills udevd: killproc udevd

--------------------------------------------------------------------------------
3.2  Controlling the rc.udev Script
--------------------------------------------------------------------------------

  # Start udevd (usually done at boot automatically):
  /etc/rc.d/rc.udev start

  # Stop udevd:
  /etc/rc.d/rc.udev stop

  # Restart (stop + start):
  /etc/rc.d/rc.udev restart

  # Reload rules without restarting the daemon:
  /etc/rc.d/rc.udev reload
--------------------------------------------------------------------------------
3.3  Making rc.udev Executable (Enable at Boot)
--------------------------------------------------------------------------------

On Nakshatra system, a script runs at boot only if it is EXECUTABLE.
If rc.udev is not executable, udev will not start.

  # Check:
  ls -l /etc/rc.d/rc.udev

  # Enable (make it run at boot):
  chmod +x /etc/rc.d/rc.udev

--------------------------------------------------------------------------------
3.4  Manually Start udevd (Without rc.udev)
--------------------------------------------------------------------------------

  # Start the daemon directly:
  /sbin/udevd --daemon

  # Then replay add events for currently connected hardware:
  udevadm trigger --action=add

  # Wait for all events to finish:
  udevadm settle

  # Check it's running:
  pidof udevd

  # Stop it:
  kill $(pidof udevd)

================================================================================
4.  udevadm — THE PRIMARY TOOL
================================================================================

udevadm is at /sbin/udevadm.

--------------------------------------------------------------------------------
4.1  Subcommand Reference
--------------------------------------------------------------------------------

  udevadm info          Query the udev database or sysfs for a device
  udevadm trigger       Re-fire uevent(s) from the kernel (simulate hotplug)
  udevadm settle        Block until all queued uevents are processed
  udevadm monitor       Stream kernel and udev events to stdout in real time
  udevadm test <sysfs-device-path>         Dry-run rule processing for a device — no side effects


--------------------------------------------------------------------------------
4.2  Querying Devices
--------------------------------------------------------------------------------

  # Show all properties of a block device:
  udevadm info --query=all --name=/dev/sda

  # Show all properties by sysfs path:
  udevadm info --path=/sys/class/block/sda

  # Export only KEY=VALUE pairs (useful in scripts):
  udevadm info --query=property --name=/dev/sda

  # Walk the attribute tree — use this to find ATTR{} values for rules:
  udevadm info --attribute-walk --name=/dev/sda

  # Example of attribute-walk output (directly usable in rules):
  #   KERNEL=="sda"
  #   SUBSYSTEM=="block"
  #   ATTR{size}=="976773168"
  #   ATTRS{serial}=="WD-WCC6Y4XXXXXXX"
  #   ATTRS{vendor}=="ATA     "
  #   ATTRS{model}=="WDC WD5000AAKX"

  # Find the ID_SERIAL for disk renaming rules:
  udevadm info --query=property --name=/dev/sda | grep ID_SERIAL
  # ID_SERIAL=WDC_WD5000AAKX-001CA0_WD-WCC6Y4XXXXXXX

  # Find USB device attributes:
  udevadm info --attribute-walk --name=/dev/bus/usb/001/001

--------------------------------------------------------------------------------
4.3  Reloading Rules After Editing
--------------------------------------------------------------------------------

  # reload rules:
  udevadm control --reload-rules

  # Then re-trigger all block devices:
  udevadm trigger --subsystem-match=block

  # Or trigger everything:
  udevadm trigger

  # Wait for queue to drain:
  udevadm settle

  # Combined one-liner (use this after every rule change):
  udevadm control --reload-rules && udevadm trigger && udevadm settle

--------------------------------------------------------------------------------
4.4  Testing Rules (Dry Run — No Side Effects)
--------------------------------------------------------------------------------

  # Get the sysfs path first:
  udevadm info --name=/dev/sda | grep '^P:'
  # P: /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda

  # Dry-run all rules against that device:
  udevadm test /sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda

  # Output shows what rules matched and what actions would be taken:
  # read rules file '/lib/udev/rules.d/60-persistent-storage.rules'
  # read rules file '/etc/udev/rules.d/99-my-disk.rules'
  # LINK 'disk/my-data-drive' /dev/sda
  # TAGS ':my_tag:'

  # IMPORTANT: Prefix the path with /sys when passing to udevadm test

--------------------------------------------------------------------------------
4.5  Monitoring Hardware Events
--------------------------------------------------------------------------------

  # Watch all events in real time (open a second terminal while doing this):
  udevadm monitor

  # Watch with full property dump:
  udevadm monitor --kernel --udev --property

  # Filter by subsystem:
  udevadm monitor --subsystem-match=block
  udevadm monitor --subsystem-match=usb
  udevadm monitor --subsystem-match=net
  udevadm monitor --subsystem-match=input

  # Filter by tag:
  udevadm monitor --tag-match=my_custom_tag

  # To generate an event to watch: plug/unplug hardware, or:
  udevadm trigger --action=change --sysname-match=sda

--------------------------------------------------------------------------------
4.6  Triggering Events
--------------------------------------------------------------------------------

  # Replay add events for all current hardware (done at boot by rc.udev):
  udevadm trigger --action=add

  # Trigger only block devices:
  udevadm trigger --subsystem-match=block

  # Trigger a specific device:
  udevadm trigger --sysname-match=sda

  # Trigger with change action (for existing devices):
  udevadm trigger --action=change --sysname-match=sda

  # Dry run — see what would be triggered without doing it:
  udevadm trigger --dry-run

================================================================================
5.  WRITING udev RULES.s
================================================================================

Rules are identical in syntax to upstream udev. The only difference is the
init system used to reload and trigger them.

--------------------------------------------------------------------------------
5.1  Where to Put Your Rules
--------------------------------------------------------------------------------

  CORRECT location:   /etc/udev/rules.d/
  WRONG location:     /lib/udev/rules.d/  (overwritten by upgradepkg!)

  File naming convention (processed in alphabetical order):
  90-99 prefix = your custom rules, processed LAST (highest priority)

  Examples:
    /etc/udev/rules.d/90-disk-names.rules
    /etc/udev/rules.d/91-usb-permissions.rules
    /etc/udev/rules.d/99-local.rules

  Permissions on your rules files:
    chmod 644 /etc/udev/rules.d/99-my-rule.rules
    chown root:root /etc/udev/rules.d/99-my-rule.rules

--------------------------------------------------------------------------------
5.2  Rule Syntax
--------------------------------------------------------------------------------

  # Match operators
  KEY=="value"       equals
  KEY!="value"       not equals
  KEY=="val*"        glob wildcard
  KEY=="val1|val2"   OR match

  # Assignment operators
  KEY="value"        assign
  KEY+="value"       append  (use for SYMLINK, TAG, RUN)
  KEY-="value"       remove from list
  KEY:="value"       assign and lock (cannot be changed by later rules)

  # Full example:
  SUBSYSTEM=="block", KERNEL=="sda", ACTION=="add", \
    SYMLINK+="disk/my-disk", TAG+="my_tag", \
    RUN+="/usr/local/bin/on-disk-added.sh"

--------------------------------------------------------------------------------
5.3  Match Keys
--------------------------------------------------------------------------------

  ACTION          Event type: add, remove, change, bind, unbind
  SUBSYSTEM       block, net, usb, input, tty, sound, ...
  KERNEL          Device kernel name: sda, sdb, eth0, ttyUSB0, ...
  DEVTYPE         disk, partition, usb_device, usb_interface, ...
  DRIVER          Kernel driver name
  ATTR{x}         sysfs attribute on THIS device
  ATTRS{x}        sysfs attribute on this device OR any parent
  ENV{x}          udev environment variable
  TAG             Match devices tagged with a specific tag
  PROGRAM         Run external program; rule matches if exit code is 0
  RESULT          Match against output of last PROGRAM

--------------------------------------------------------------------------------
5.4  Assignment Keys
--------------------------------------------------------------------------------

  NAME            Rename the /dev node (risky — prefer SYMLINK)
  SYMLINK+=       Create a persistent symlink under /dev
  OWNER           Set device node owner
  GROUP           Set device node group
  MODE            Set permissions (octal: 0660, 0664, 0600...)
  ENV{x}=         Set an environment variable on the device
  TAG+=           Tag the device for later querying
  RUN+=           Run a script when the rule fires
  LABEL=          Define a target for GOTO
  GOTO=           Jump to a LABEL
  OPTIONS+=       Set options: watch, static_node, string_escape...

--------------------------------------------------------------------------------
5.5  Substitution Specifiers
--------------------------------------------------------------------------------

  %k  / $kernel      Kernel name (e.g. sda, sdb1, ttyUSB0)
  %n  / $number      Device number (e.g. partition number)
  %p  / $devpath     Full sysfs path
  %E{VAR}            Value of udev env variable VAR
  %s{attr}           Value of sysfs attribute
  %M  / $major       Major device number
  %m  / $minor       Minor device number

================================================================================
6.  DISK RENAMING AND PERSISTENT SYMLINKS
================================================================================

60-persistent-storage.rules already creates:
  /dev/disk/by-id/     — by hardware serial/model
  /dev/disk/by-uuid/   — by filesystem UUID
  /dev/disk/by-label/  — by filesystem label
  /dev/disk/by-path/   — by hardware connection path

You add your OWN custom symlinks on top using SYMLINK+=

--------------------------------------------------------------------------------
6.1  Find Disk Identifiers (Before Writing Rules)
--------------------------------------------------------------------------------

  # Method 1: udevadm — find ID_SERIAL
  udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL|ID_MODEL|DEVLINKS'

  # Method 2: list the existing by-id symlinks
  ls -la /dev/disk/by-id/

  # Method 3: blkid (shows UUIDs and labels)
  blkid

  # Method 4: Walk all attributes
  udevadm info --attribute-walk --name=/dev/sda | grep -E 'serial|model|vendor'

  # Example output:
  # ID_SERIAL=WDC_WD5000AAKX-001CA0_WD-WCC6Y4XXXXXXX
  # ID_MODEL=WDC_WD5000AAKX-001CA0
  # DEVLINKS=/dev/disk/by-id/ata-WDC_WD5000... /dev/disk/by-path/pci-...

--------------------------------------------------------------------------------
6.2  Create a Custom Persistent Symlink by Serial Number
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/90-disk-names.rules

  # Rename a specific disk by serial — stable even if sda/sdb changes:
  SUBSYSTEM=="block", ENV{ID_SERIAL}=="WDC_WD5000AAKX-001CA0_WD-WCC6Y4XXXXXXX", \
    SYMLINK+="disk/my-data-drive"

  # Rename a specific disk by model (matches all disks of that model):
  SUBSYSTEM=="block", ENV{ID_MODEL}=="WDC_WD5000AAKX-001CA0", \
    SYMLINK+="disk/western-digital"

  # Rename by UUID (most stable — survives disk swaps):
  # First find the UUID: blkid /dev/sda1
  SUBSYSTEM=="block", ENV{ID_FS_UUID}=="a1b2c3d4-e5f6-7890-abcd-ef1234567890", \
    SYMLINK+="disk/root-partition"

  # Reload and apply:
  udevadm control --reload-rules && udevadm trigger && udevadm settle

  # Confirm symlink created:
  ls -la /dev/disk/my-data-drive
  # lrwxrwxrwx 1 root root 3 /dev/disk/my-data-drive -> ../sda

--------------------------------------------------------------------------------
6.3  Create Symlink by Kernel Name (Simpler But Less Stable)
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/90-disk-names.rules

  SUBSYSTEM=="block", KERNEL=="sda", SYMLINK+="disk/primary-disk"
  SUBSYSTEM=="block", KERNEL=="sdb", SYMLINK+="disk/secondary-disk"

  # NOTE: sda/sdb order can change between reboots if you have multiple disks.
  # Always prefer serial-based or UUID-based matching for stability.

--------------------------------------------------------------------------------
6.4  Tag Disks for Scripting
--------------------------------------------------------------------------------

  SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", TAG+="my_harddisks"
  SUBSYSTEM=="block", KERNEL=="sd*[0-9]",  TAG+="my_partitions"

  # Query all tagged devices:
  udevadm trigger --tag-match=my_harddisks --dry-run

--------------------------------------------------------------------------------
6.5  Rename Network Interfaces
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/70-net-names.rules
  # Rename by MAC address (survives driver changes):
  SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="lan0"

  # Rename by PCI slot:
  SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:03:00.0", NAME="eth-external"

  # Apply — requires reboot or hotplug event for network interfaces:
  udevadm control --reload-rules

================================================================================
7.  PERMISSIONS, OWNERSHIP, GROUPS
================================================================================

--------------------------------------------------------------------------------
7.1  Serial Port / USB Serial Access
--------------------------------------------------------------------------------

  # Add your user to the 'dialout' group:
  usermod -a -G dialout tushar

  # Verify:
  groups tushar

  # Custom rule to ensure ttyUSB0 is accessible by dialout group:
  # /etc/udev/rules.d/91-serial-permissions.rules
  SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]*", MODE="0660", GROUP="dialout"
  SUBSYSTEM=="tty", KERNEL=="ttyS[0-9]*",   MODE="0660", GROUP="dialout"

--------------------------------------------------------------------------------
7.2  USB Device Permissions
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/91-usb-permissions.rules

  # All USB devices — allow plugdev group:
  SUBSYSTEM=="usb", MODE="0664", GROUP="plugdev"

  # Specific device by Vendor:Product ID:
  SUBSYSTEM=="usb", ATTRS{idVendor}=="046d", ATTRS{idProduct}=="c52b", \
    MODE="0664", GROUP="plugdev"

  # Finding Vendor:Product ID:
  lsusb
  # Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver
  udevadm info --attribute-walk --name=/dev/bus/usb/001/003 | grep -E 'idVendor|idProduct'

--------------------------------------------------------------------------------
7.3  Block Device Access for Non-Root Users
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/91-block-permissions.rules

  # Allow disk group to access loop devices:
  SUBSYSTEM=="block", KERNEL=="loop[0-9]*", MODE="0660", GROUP="disk"

  # Allow a specific user to access a specific disk:
  SUBSYSTEM=="block", ENV{ID_SERIAL}=="MY_DISK_SERIAL", OWNER="tushar"

================================================================================
8.  HARDWARE EVENT MONITORING
================================================================================

--------------------------------------------------------------------------------
8.1  Real-Time Event Stream
--------------------------------------------------------------------------------

  # Open this in a terminal, then plug/unplug hardware:
  udevadm monitor --kernel --udev --property

  # Only block device events:
  udevadm monitor --subsystem-match=block

  # Only USB events:
  udevadm monitor --subsystem-match=usb

  # What you see when a USB drive is plugged in:
  # KERNEL[142.887] add /devices/pci.../usb1/1-1 (usb)
  # ACTION=add
  # DEVPATH=/devices/pci.../usb1/1-1
  # SUBSYSTEM=usb
  # DEVTYPE=usb_device
  # ...
  # KERNEL[142.910] add /devices/pci.../usb1/1-1/1-1:1.0/host5/... (block)
  # ACTION=add
  # DEVNAME=/dev/sdb
  # DEVTYPE=disk

--------------------------------------------------------------------------------
8.2  RUN Scripts Notes
--------------------------------------------------------------------------------

Since Nakshatra system does NOT have systemd, you must use RUN+= scripts OR
write your own rc.d-style handler. There is no systemd device unit available.

  # /etc/udev/rules.d/90-usb-events.rules
  SUBSYSTEM=="block", ACTION=="add", SUBSYSTEMS=="usb", \
    RUN+="/usr/local/bin/usb-disk-added.sh"

  SUBSYSTEM=="block", ACTION=="remove", SUBSYSTEMS=="usb", \
    RUN+="/usr/local/bin/usb-disk-removed.sh"

  # /usr/local/bin/usb-disk-added.sh
  #!/bin/sh
  # udev env vars are available: $DEVNAME, $ACTION, $SUBSYSTEM, etc.
  LOGFILE="/var/log/udev-usb-events.log"
  echo "$(date): USB disk ADDED: $DEVNAME  (type=$DEVTYPE)" >> $LOGFILE

  # /usr/local/bin/usb-disk-removed.sh
  #!/bin/sh
  LOGFILE="/var/log/udev-usb-events.log"
  echo "$(date): USB disk REMOVED: $DEVNAME" >> $LOGFILE

  # Make scripts executable:
  chmod +x /usr/local/bin/usb-disk-added.sh
  chmod +x /usr/local/bin/usb-disk-removed.sh

RULES FOR RUN+= SCRIPTS:
  - Must be an absolute path
  - No shell syntax inside the rule (pipes, redirects, $(cmd)) — use a script
  - The script receives ALL udev env vars ($DEVNAME, $ACTION, $MAJOR, etc.)
  - Keep it fast — slow scripts block the udev event queue
  - Background long tasks:  RUN+="/usr/local/bin/my-script.sh &"


================================================================================
9.  HARDWARE DATABASE (hwdb)
================================================================================

--------------------------------------------------------------------------------
9.1  Hardware Database Files. 
--------------------------------------------------------------------------------

  # System hwdb (distro-supplied, do not edit):
  /lib/udev/hwdb.d/*.hwdb

  # Custom hwdb (your entries go here):
  /etc/udev/hwdb.d/70-my-custom.hwdb

  # hwdb format — match string followed by KEY=value:
  evdev:input:b0003v046DpC52B*
   KEYBOARD_KEY_70039=capslock

  # After any hwdb change, rebuild the binary db:
  udevadm hwdb --update

  # Then trigger input devices to re-read:
  udevadm trigger --subsystem-match=input

--------------------------------------------------------------------------------
9.2  Query hwdb
--------------------------------------------------------------------------------

  udevadm hwdb --test="block:svirtio_blk:v00001AF4p00001001:"
  udevadm hwdb --test="input:b0003v046DpC52B*"


================================================================================
10.  ADVANCED PATTERNS
================================================================================

--------------------------------------------------------------------------------
10.1  Multi-Line Rules
--------------------------------------------------------------------------------

  SUBSYSTEM=="block", \
    KERNEL=="sd*[!0-9]", \
    ENV{ID_VENDOR}=="WDC", \
    TAG+="wd_disks", \
    SYMLINK+="disk/wd-%k", \
    RUN+="/usr/local/bin/disk-added.sh"

--------------------------------------------------------------------------------
10.2  GOTO / LABEL for Early Exit
--------------------------------------------------------------------------------

  # Skip virtual/loop devices entirely, don't apply any rules to them:
  SUBSYSTEM=="block", KERNEL=="loop*", GOTO="end_of_block_rules"

  SUBSYSTEM=="block", KERNEL=="sd*", SYMLINK+="disk/real-disk-%n"
  SUBSYSTEM=="block", ENV{ID_VENDOR}=="WDC", TAG+="wd_disk"

  LABEL="end_of_block_rules"

--------------------------------------------------------------------------------
10.3  PROGRAM and RESULT
--------------------------------------------------------------------------------

  # Run a program; rule matches only if it exits 0:
  SUBSYSTEM=="block", PROGRAM=="/usr/local/bin/check-disk.sh %k", \
    SYMLINK+="disk/verified-%k"

  # Capture output and use it in assignments:
  SUBSYSTEM=="block", KERNEL=="sd*", \
    PROGRAM=="/lib/udev/scsi_id -u -g $devnode", \
    ENV{SCSI_ID}="$result", \
    SYMLINK+="disk/by-scsi-$env{SCSI_ID}"

--------------------------------------------------------------------------------
10.4  IMPORT
--------------------------------------------------------------------------------

  # Import properties from helper program (common for SCSI/ATA):
  SUBSYSTEM=="block", IMPORT{program}="/lib/udev/ata_id --export $devnode"
  SUBSYSTEM=="block", IMPORT{program}="/lib/udev/scsi_id --export --whitelisted $devnode"

  # Import from parent device:
  SUBSYSTEM=="block", DEVTYPE=="partition", \
    IMPORT{parent}="ID_[^F]*"

--------------------------------------------------------------------------------
10.5  Full USB Storage Device Rule
--------------------------------------------------------------------------------

  # /etc/udev/rules.d/90-sandisk-usb.rules
  ACTION=="add", \
    SUBSYSTEM=="block", \
    SUBSYSTEMS=="usb", \
    ATTRS{idVendor}=="0781", \
    ATTRS{idProduct}=="5583", \
    SYMLINK+="disk/sandisk-backup", \
    MODE="0660", \
    GROUP="plugdev", \
    TAG+="backup_drive", \
    RUN+="/usr/local/bin/mount-backup.sh"

================================================================================
11.  DEBUGGING 
================================================================================

--------------------------------------------------------------------------------
11.1  Step-by-Step Debug Workflow 
--------------------------------------------------------------------------------

  1. Write your rule file in /etc/udev/rules.d/99-test.rules
     chmod 644 /etc/udev/rules.d/99-test.rules

  2. Verify syntax (eudev 3.2+):
     udevadm verify /etc/udev/rules.d/99-test.rules

  3. Find the sysfs path of your target device:
     udevadm info --name=/dev/sda | grep '^P:'
     # P: /devices/pci0000:00/0000:00:1f.2/.../block/sda

  4. Dry-run the rules (prefix path with /sys):
     udevadm test /sys/devices/pci0000:00/0000:00:1f.2/.../block/sda

  5. Reload rules:
     udevadm control --reload-rules

  6. Trigger the device:
     udevadm trigger --sysname-match=sda && udevadm settle

  7. Check the result:
     udevadm info --query=all --name=/dev/sda
     ls -la /dev/disk/

  8. Check daemon logs —  (no journald), use dmesg and syslog:
     dmesg | grep -i udev
     grep udev /var/log/syslog 2>/dev/null
     grep udev /var/log/messages

--------------------------------------------------------------------------------
11.2  Enable udevd Debug Logging
--------------------------------------------------------------------------------

  # Enable debug level:
  udevadm control --log-level=debug

  # Watch log output (check where syslog goes on your system):
  tail -f /var/log/syslog
  # or:
  tail -f /var/log/messages

  # Reset to normal:
  udevadm control --log-level=info

  # Alternatively, start udevd in foreground for debugging:
  killall udevd 2>/dev/null
  /sbin/udevd --debug 2>&1 | tee /tmp/udevd-debug.log &

--------------------------------------------------------------------------------
11.3  Common Mistakes 
--------------------------------------------------------------------------------

  MISTAKE                                FIX
  -----------------------------------    ----------------------------------------
  Using systemctl to control udevd       Use /etc/rc.d/rc.udev restart
                                         or: udevadm control --reload-rules

  Rules in /lib/udev/rules.d/            Use /etc/udev/rules.d/ only
  (gets overwritten by upgradepkg)

  rc.udev not executable                 chmod +x /etc/rc.d/rc.udev

  RUN+= with inline shell $(cmd)         Use a separate wrapper script

  Forgetting to reload rules             udevadm control --reload-rules
  after editing                          && udevadm trigger

  Rule file not 644 / owned by root      chmod 644 && chown root:root

  Matching ATTRS from wrong parent       ATTRS only works ONE parent level up
                                         per rule; check with --attribute-walk

  NAME= on block device                  Use SYMLINK+= instead. NAME= can
                                         break /dev/disk/ links.

  No journalctl available                Use dmesg | grep udev
                                         or tail -f /var/log/messages

--------------------------------------------------------------------------------
11.4  Get sysfs Path for a Device
--------------------------------------------------------------------------------

  # Method 1:
  udevadm info --name=/dev/sda | grep '^P:'
  # P: /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda

  # Method 2: readlink through /sys/class
  readlink -f /sys/class/block/sda

  # udevadm test requires the path WITHOUT the /sys prefix in some versions,
  # and WITH /sys in others. Try both:
  udevadm test /sys/devices/pci0000:00/0000:00:1f.2/.../block/sda
  udevadm test /devices/pci0000:00/0000:00:1f.2/.../block/sda

--------------------------------------------------------------------------------
11.5  Check What Rules Were Applied to a Device
--------------------------------------------------------------------------------

  # Export full database entry for a device:
  udevadm info --export-db | grep -A 20 "^N:sda$"

  # List all symlinks for a device:
  udevadm info --query=symlink --name=/dev/sda

  # List all properties:
  udevadm info --query=property --name=/dev/sda

================================================================================
12.  QUICK REFERENCE CARD 
================================================================================

--------------------------------------------------------------------------------
12.1  Daemon Control
--------------------------------------------------------------------------------

  Start udevd              :  /etc/rc.d/rc.udev start
                              # or: /sbin/udevd --daemon
  Stop udevd               :  /etc/rc.d/rc.udev stop
  Restart udevd            :  /etc/rc.d/rc.udev restart
  Reload rules             :  udevadm control --reload-rules
  After rule change        :  udevadm control --reload-rules && udevadm trigger
  Enable at boot           :  chmod +x /etc/rc.d/rc.udev
  Disable at boot          :  chmod -x /etc/rc.d/rc.udev
  Check running            :  pidof udevd   OR   /etc/rc.d/rc.udev status

--------------------------------------------------------------------------------
12.2  Query Commands
--------------------------------------------------------------------------------

  All properties           :  udevadm info --query=all --name=/dev/sda
  Attribute walk           :  udevadm info --attribute-walk --name=/dev/sda
  Key=value only           :  udevadm info --query=property --name=/dev/sda
  Symlinks only            :  udevadm info --query=symlink --name=/dev/sda
  By sysfs path            :  udevadm info --path=/sys/class/block/sda
  Serial of disk           :  udevadm info -q property -n /dev/sda | grep ID_SERIAL
  USB vendor/product       :  udevadm info -a -n /dev/bus/usb/001/003 | grep idVendor

--------------------------------------------------------------------------------
12.3  Trigger & Settle
--------------------------------------------------------------------------------

  Trigger all              :  udevadm trigger
  Trigger block only       :  udevadm trigger --subsystem-match=block
  Trigger one device       :  udevadm trigger --sysname-match=sda
  Dry run                  :  udevadm trigger --dry-run
  Wait for queue           :  udevadm settle
  Wait with timeout        :  udevadm settle --timeout=10

--------------------------------------------------------------------------------
12.4  Rule File Cheat Sheet
--------------------------------------------------------------------------------

  # Custom symlink by kernel name:
  SUBSYSTEM=="block", KERNEL=="sda", SYMLINK+="disk/my-primary"

  # Custom symlink by serial number (preferred — stable):
  SUBSYSTEM=="block", ENV{ID_SERIAL}=="YOUR_SERIAL_HERE", SYMLINK+="disk/my-disk"

  # Custom symlink by UUID:
  SUBSYSTEM=="block", ENV{ID_FS_UUID}=="YOUR-UUID-HERE", SYMLINK+="disk/root-part"

  # Set permissions on serial port:
  SUBSYSTEM=="tty", KERNEL=="ttyUSB0", MODE="0660", GROUP="dialout"

  # USB device by vendor:product:
  SUBSYSTEM=="usb", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", \
    MODE="0664", GROUP="plugdev"

  # Run script on device add:
  SUBSYSTEM=="block", ACTION=="add", RUN+="/usr/local/bin/on-add.sh"

  # Tag a device:
  SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", TAG+="my_disks"

  # Skip virtual devices:
  SUBSYSTEM=="block", KERNEL=="loop*", GOTO="end"
  LABEL="end"

  # Network interface rename by MAC:
  SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="lan0"

--------------------------------------------------------------------------------
12.5  Key Files and Paths 
--------------------------------------------------------------------------------

  /etc/rc.d/rc.udev              Init script (start/stop/reload udevd)
  /sbin/udevd                    eudev daemon binary
  /sbin/udevadm                  Management tool
  /etc/udev/udev.conf            Main config (udev_root, log level)
  /etc/udev/rules.d/             YOUR custom rules (persistent)
  /lib/udev/rules.d/             System rules (do not edit)
  /lib/udev/                     Helper binaries (ata_id, scsi_id, etc.)
  /run/udev/data/                Runtime device database
  /var/log/packages/eudev-*      Package install log (list all eudev files)


================================================================================
13 Conclusion:
================================================================================
udev is the device manager responsible for detecting hardware, creating device nodes under /dev, applying permissions, generating persistent symlinks, and executing actions when devices are added, removed, or changed. It operates through the udevd daemon, which processes kernel-generated uevents according to rules defined in /lib/udev/rules.d/ (system rules) and /etc/udev/rules.d/ (custom rules).

The most important administration tool is udevadm, which is used to query device information, monitor hardware events, test rules, reload configurations, and trigger device events. Custom rules should always be placed in /etc/udev/rules.d/ and typically use SYMLINK+= instead of renaming device nodes directly. Persistent device identification is best achieved using serial numbers (ID_SERIAL) or filesystem UUIDs (ID_FS_UUID), as kernel-assigned names such as sda and sdb can change between boots.

On systems without systemd, eudev is managed through /etc/rc.d/rc.udev, which starts the daemon during boot, triggers device discovery, and ensures the device database is populated. Proper debugging involves validating rules, testing them with udevadm test, monitoring events with udevadm monitor, and checking system logs.
================================================================================