So in putty at 9600, I initially saw repeated strings such as this. If I use a keyfob to trigger a relay, the last 2 digits change to 80 or 81
0986006000C8606314C05181
DL900 still isn't working, not sure why, but it looks like I may be making some progress after power cycling the panel.
http://diysecurityforum.com/index.php?topic=19054.0 had someone with a similar truncated message issue in DL900.
At this point, I'm no longer hanging, it's just failing:
01 10/12/12 21:48:51.332 ^[[31;1mLuImplementation::StartLua running startup code for 13 I_CaddxNX584Security.xml failed^[[0m <0x2be4d680>
31 10/12/12 21:48:51.333 AlarmManager::Run finish callback for alarm 0xdce2e0 entry 0xe1b768 type 5 id 14 param=0xe0bb10 entry->when: 1350103691 time: 13
31 10/12/12 21:48:51.333 AlarmManager::Run callback for alarm 0xdce2e0 entry 0xe08b08 type 158 id 28 param=(nil) entry->when: 1350103694 time: 1350103731
02 10/12/12 21:48:51.334 ^[[33;1mUserData::AlarmCallback ALARM_RESYNC_DEVICES^[[0m <0x2be4d680>
10 10/12/12 21:48:51.338 FileUtils::ReadURL starting
https://cms2.mios.com/sync_device?PK_AccessPoint=30003868&HW_Key=<redacted for security> <0
10 10/12/12 21:48:51.339 XXX-UpdateSystemMessagesTasks now 2=Caddx NX584 Security System[13]: Failed to set up interface <0x2bc4d680>
I double checked the settings, and was missing the set calendar command. I fixed that, reset everything, and now it looks like it's just rejecting all bytes received:
50 10/12/12 23:03:48.273 luup_log:13: Initializing Caddx NX-584 __LEAK__ this:126976 start:299008 to 0x8b6000 <0x2b7db680>
50 10/12/12 23:03:48.273 luup_log:13: Opening serial port <0x2b7db680>
50 10/12/12 23:03:48.274 luup_log:13: Sending message and waiting for response: 0x21 Interface Configuration Request <0x2b7db680>
50 10/12/12 23:03:48.275 luup_log:13: Message: Outgoing: 0x7e 0x01 0x21 0x22 0x23 <0x2b7db680>
02 10/12/12 23:03:48.279 ^[[33;1mIOPort::Run RecvFailed 0 close 12^[[0m <0x2d9db680>
02 10/12/12 23:03:53.302 ^[[33;1mIOPort::Run RecvFailed 0 close 12^[[0m <0x2d9db680>
50 10/12/12 23:03:58.575 luup_log:13: Ignoring byte 0a <0x2b7db680>
50 10/12/12 23:03:58.586 luup_log:13: Ignoring byte 30 <0x2b7db680>
50 10/12/12 23:03:58.596 luup_log:13: Ignoring byte 42 <0x2b7db680>
50 10/12/12 23:03:58.607 luup_log:13: Ignoring byte 38 <0x2b7db680>
50 10/12/12 23:03:58.617 luup_log:13: Ignoring byte 31 <0x2b7db680>
50 10/12/12 23:03:58.628 luup_log:13: Ignoring byte 33 <0x2b7db680>
50 10/12/12 23:03:58.638 luup_log:13: Ignoring byte 35 <0x2b7db680>
50 10/12/12 23:03:58.649 luup_log:13: Ignoring byte 32 <0x2b7db680>
50 10/12/12 23:03:58.659 luup_log:13: Ignoring byte 45 <0x2b7db680>
50 10/12/12 23:03:58.670 luup_log:13: Ignoring byte 33 <0x2b7db680>
50 10/12/12 23:03:58.680 luup_log:13: Ignoring byte 33 <0x2b7db680>
50 10/12/12 23:03:58.690 luup_log:13: Ignoring byte 33 <0x2b7db680>
Last update in this post, I need to get some sleep

Ignoring byte 0a.... 0d - those look familiar. It looks like I needed to be in binary.
I set the panel to binary, 9600, then power cycled the panel for good measure. Once the panel was up, I restarted luup, and got much further. It recognized the capabilities, but appears to have hung attempting to set the time or shortly thereafter. It eventually failed. This log is larger, so I have attached it.
I think it's getting very close. I'll be sure to update the wiki page with the NX-8e specifics (locations & segments to set, photo of my cable, serial settings) once its working.