Harden serial ioctl user-pointer handling in RT-Smart

[Copilot]

RT-Smart was forwarding serial ioctl() payload pointers from user space directly into kernel/driver code. That let a user process trigger unchecked dereferences in the serial core and pass invalid serial config into BSP UART code, including a concrete baud_rate == 0 divide-by-zero path on imx6ull-smart.

• Marshal serial ioctl payloads at the fd boundary

  • Update serial POSIX ioctl handling in both serial v1 and v2 to copy known user payloads into kernel memory before calling rt_device_control().
  • Copy result-bearing payloads back to user space for read-style ioctls.
  • Keep unknown commands on the existing fast path instead of adding a generic syscall-wide marshal layer.

• Block user-space callback injection

  • Reject RT_DEVICE_CTRL_NOTIFY_SET from LWP/user-space serial fds so user pointers cannot be installed as kernel callback state.

• Reject invalid serial configuration earlier

  • Return -RT_EINVAL for baud_rate == 0 in the serial core before config is applied.
  • Apply hardware configuration before committing serial->config, so failed reconfiguration does not leave partially updated state.

• Close the imx6ull-smart crash path

  • In the imx6ull-smart UART driver, reject zero or out-of-range baud rates before programming divider registers.

Example of the affected path after this change:

case RT_DEVICE_CTRL_CONFIG:
{
    struct serial_configure kcfg;

    if (!lwp_user_accessable(args, sizeof(kcfg)) ||
        lwp_get_from_user(&kcfg, args, sizeof(kcfg)) != sizeof(kcfg))
    {
        return -RT_EFAULT;
    }

    if (kcfg.baud_rate == 0)
    {
        return -RT_EINVAL;
    }

    return rt_device_control(device, cmd, &kcfg);
}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

📌 Code Review Assignment

🏷️ Tag: components

Reviewers: @Maihuanyi

Changed Files (Click to expand)

• components/drivers/serial/dev_serial.c
• components/drivers/serial/dev_serial_v2.c

🏷️ Tag: components_driver_serial_v2

Reviewers: @Ryan-CW-Code

Changed Files (Click to expand)

• components/drivers/serial/dev_serial_v2.c

---

📊 Current Review Status (Last Updated: 2026-06-19 05:22 CST)

• ⌛ @Maihuanyi Pending Review
• ⌛ @Ryan-CW-Code Pending Review

---

📝 Review Instructions

1. 维护者可以通过单击此处来刷新审查状态: 🔄 刷新状态
   Maintainers can refresh the review status by clicking here: 🔄 Refresh Status

2. 确认审核通过后评论 LGTM/lgtm
   Comment LGTM/lgtm after confirming approval

3. PR合并前需至少一位维护者确认
   PR must be confirmed by at least one maintainer before merging

ℹ️ 刷新CI状态操作需要具备仓库写入权限。
ℹ️ Refresh CI status operation requires repository Write permission.