We have moved at community.getvera.com

Author Topic: lua https command response_body encrypted  (Read 1857 times)

Offline joer.

  • Full Member
  • ***
  • Posts: 128
  • Karma: +1/-0
lua https command response_body encrypted
« on: September 02, 2015, 11:32:48 pm »
So guys,
I have a project where I log into my OWA (Company email), scrape the page, and have my email read to me in the morning while I have breakfast. Something to that nature.

I have actually figured out a lot so far, when it comes to making https post calls, parsing the correct cookies, and dealing with the redirects.  Thanks to Postman, and luatester.

I am stuck on something now I can login and make a successful call however the response body is always encrypted.  For the point of this post I have pruned everything down to a simple Google call.

Code: [Select]
local https = require "ssl.https"
local ltn12 = require "ltn12"
local util = require "util"

local response_body = { }

local res, code, response_headers = https.request
{
  url = "https://www.google.com";
  method = "GET";
  protocol = "tlsv1";
  headers =
  {
    ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
    ["User-Agent"] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36";
    ["Accept-Encoding"] = "gzip, deflate";
    ["Accept-Language"] = "en-US,en;q=0.8";
    ["Content-Length"] = 0;
  };
  sink = ltn12.sink.table(response_body);
}


print("Status:", res and "OK" or "FAILED")
print("HTTP code:", code)
print("Response headers:")
print(response_headers)
if type(response_headers) == "table" then
  for k, v in pairs(response_headers) do
    print(k, ":", v)
  end
else
  -- Would be nil, if there is an error
  print("Not a table:", type(response_headers))
end

print("Response body:")
if type(response_body) == "table" then
  print(table.concat(response_body))
else
  -- Would be nil, if there is an error
  print("Not a table:", type(response_body))
end
print("Done dumping response")

Most of this has been from googling around and I admit I still don't have a great grasp. The output is very hopeful. A successful 200 call with response headers that are legible.  It is the response_body which is jibberish.

Response:
Code: [Select]
Print output
Status:     OK     
HTTP code:     200     
Response headers:     
table: 0x15eb3d8     
set-cookie     :     PREF=ID=1111111111111111:FF=0:TM=1441251094:LM=1441251095:V=1:S=b1ID5JBw3l8ZTjUz; expires=Thu, 31-Dec-2015 16:02:17 GMT; path=/; domain=.google.com, NID=71=2JsP70BMZBg2TyiNBrAmbYd0CNMEtwl6IEzVywtPQecbQIz_8vtad0kvz9pXkRZwql42SMfp4ghU6AbYpTURSCMi8heuJTHUuN4_pr3JeRiIZ9XDCEznTR3OWSLE-ly9; expires=Fri, 04-Mar-2016 03:31:35 GMT; path=/; domain=.google.com; HttpOnly     
p3p     :     CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."     
date     :     Thu, 03 Sep 2015 03:31:34 GMT     
x-xss-protection     :     1; mode=block     
content-encoding     :     gzip     
transfer-encoding     :     chunked     
expires     :     -1     
cache-control     :     private, max-age=0     
connection     :     close     
alt-svc     :     quic=":443"; p="1"; ma=604800     
alternate-protocol     :     443:quic,p=1     
x-frame-options     :     SAMEORIGIN     
content-type     :     text/html; charset=UTF-8     
server     :     gws     
Response body:     
????i[?(?}????N6&???? @??q?????8??????oi?-??????s????J?R?T?*???%?V?????{???7cz??U??.g?????kk????`;??+]9F==???'g?a0?H???@?PF???>???????zfj;????C?v?3???5??a/3?`t!m@?S?m7??Zf`;z!?m?= ?Enf??/w??&?2?????????uk?y??Är? ????????S?;z??'\!PQ?f?h ?tWu??)o???????? ?5J???|?GH??I3???? 9?j[?? ??0?(?G?=?? `???S???F
j????f?v?? ???8?%g??x???@R?A?c??!T?????M7t??(?????I??c?>e2??K???P??????g{Z ??oO?n????0?T6_??k?7=?0?;7??w??s} ?EQ??B?$??zI(??rQ??k?V???$Ib_??y.VD?\??:??J??2????"?V?2)W?I~ ?\??*?5Xz?ZdW??.J??^??s?L??2._??*?J%?H?BGY???z?<?J?\?b?^???H??D? LOH? ]$?T?$z%t?k?^??RE,?k???T?s??AEUz-?k?L?+WJ?\I}Q??k??WX??j?^?ev???J??BW??T??%JO??K?Z.???F?WkRM?B????Z??X?A??_???:tv??Uj???Z?? A?(?K?4??cWw????|?MIP&?bY ??G?q???PR?? ???@???\}.<??<???-*????}??>???s?/? ?R??{?2e???^?+P? ?Kn????B??%?A????????r?M?euS????VA?90\mMo9:??aFy???t?\??????c??z??g????9?P2S??,3?`?*???Z? ?/?0???3???3A+%`rC}??D????L.???A4Az?7$???e?L#??.??5? ?bu???S?B?`?fK?3?e???!?K???e#?? t/O??[??gp?y+?-??#???n?$?qz*?4?:??8?4V?9?hn*Kh ?Y?i??{6???-/?J2??%?w.???2H)?y???y???~+V?"?@?r??W(????? dEo???P??????s$k@?Zv%?kO>?r6?????Z??e????]V?K?+??b???,???i???Z6f??]?????w?k??;;?E??$ o??&z?`??y/??@AA??o??,??Ny?CS??? ??aX??>?fD{0(:??Z?<???mEY?9 ?yO ??r???<9?? T???????k????-??r???(/?{?+;? Gn??F1???????sA  ?eYJU????6?P?s? Q-??: ?8?C?;????*?6?#?U?oza4v??7zh?T?iS??[????4?3??o?` ?N??;???e??<?? @. ??
?6?>~????D????!?#???85?S?R"???%?q9???5?>Ef??????mF??\?os?????TJ????k}??-eyY) ?i8(??? J?y???50?`*??y?D??8?`? Z???*?A?
x? Vu?????)??r?o??????P
?>?y????O???)w????t??S2 :T?QL/}h??,b7????9?(???d????????k#n???Z?[?R??H?|8??Og?OUP0)KbPnJ4O??#?=??Ay\??????#??^r??????? ??5O?L???mj??.?Q???<~Zx?g????Od???/? ?`??"??.?~??D"J?"?? ? A)L??4???>?h&?8-?X?1:?????{-?j?N?M|?^_??]???I?!ã@m???p?i?(? ?IE??1??u?q??????2g?n6O&A??9/???#??=?}w??c????????k? N?-???+????a??U2?????? ???? v??}g??7]?????u??{?^?-a??=<)??)'??V?a??jY???o]?G??,?6?e??4?J??_Fz?7? > =Vj?=?M[`v:?]??M0? ????u?tT???Y?R???SY??V^2???v%m?,????C ;????x L3? "?????
?? b`????c???????DR? ???$l???P? s?k?f?x?XC41{???????C?v?k?
?
??m?0???gh??@A?p@UW??gz?????Mw?/??z?>o?vp?Yg?????/?????x????0??Z?c?0^?? !l?%??(l?t6???????qt?~??`????????????=z=;g??? (?/??Sv?=?p??0??=?????.??&@??f???????;{???y????Ds??6e?k!)I??e???/"? ??n??????>e3f???;
?W???R????c???O?(???t?????C??viP@8$?}?v??B????w?X3'XC8wD(?????LK???<??C???5?q=???? T=??????# ?zn?_?+kx?a???$??dx?g ?? :?\WS#X#s??Y M??eZ$&???Zx??? ;4?xU:??,?0?? ????????yT=?"?E?O?????#{\^.?{P??=$?H?4<(??2????lK?Po??*<?/0??mksy?$??c?R??TZ? ?"\??|\y?"<?A,8??d???
????z??>*x????r?z?\O?B?0? qh????,?O???o?$g? u?????d???ggWG??????>?????????!<UEk??kiP0? ????L)??o??y@?R/???<:8?8|?,?.V?????e??????>?????4?}??D?????\h?D?e1????
d=?0u ?/? M?A?*] Nx ?.n?J???N37??c???3?n'a? 9'????????bN/ ?s,????(??~?gaFRS ??,??G?
@???i??C?^?g?t??4????? ! ??|s ??Nf!???o}?UxV ?????"?C?rR=2???7??6!?0?s?xT?}T?<?c? e?n|?#?=?!?r?EC_Y??C0T_?A
?_~??Y?)?^(????B?r?W?? m?) 8??18?W?%?-?H>?y?s
?Xw '
?|??t??D3?+d:???eg>???9?x????;W?{?{???~uKx8 ?2M??HC?/??4{E????q?X &_?^?o?? G??u?????? ? ? -#?=??`D: _??a????',?Y??}}???wN?g?=F??<?Py??\^b??/??W?}?w?2TN?^???x???|CY?26??&x??/???{??????? C?KK?0X_B-?cA?1G??hly?? ?|?1?\z??18??????w3z?$?37? ?qQ ?O???a???'?b?3??=?Pa ?hA?V??4?????A?l???>??=?FU?U? qTPAyT?,/G??G]??8?4?Sr`??O?2?? 7-?y?6?a\=????St ? ?)????T?RMD?e?a? g??m+`?2 8?T?2???4???U!?M]d? ??P?????U?b??O?v?? ?J XH'W{????.Sl?5?u ??{)?z????e#?5??u#l?`??9???(?$??
M??? ?(?(pPa?U?]`?^??}y???-)??d??+??
?l t>^@??@??M?? J?/?V
v"Z??un?g??152??;??n???|u???s???lR??N??>?Uj???=h?????BS?f??Y ?r??w?9??%6E?U?5?w?6?d?Z$N??????%^%"2?e??p? ?Q?!?|\7^?????l??G??6???}?:g???jõy?~?M????|D.??p??p(?? ????p?`?? ???T.k?uBl ?{?Q??????&?Qa??¢???t?Z?FdD???X?D ?'s??Bg>?#:?w???2_??H
???g/X[~???????"??~??k?"%?B"?`/??(AhtyY??UC?tH??|l?F??.6.??@Hw??7x(.??'???
? ??% No8;?s???!}????????M r9Dn???|?????<?bh;??M l]`?i???5??g Y???????N*?3]=#a??>?tv????????8(?Û????w????O ??d|i? PMre }}?$? :J-?fYp???Y???>U?x+P?U???3?.????? \?j9?X????y??Dq???? ? ??w????-??~/?{??????
?zU15??:?'?[5]???H&K?+??# S???Ud?10v?i ??2p? |h? k?P4?3A?W?e??M?lAe?0A?s&????????za? ?2??y???b{?=`?M2???b?C?PA????8j2K?oZACo^?)?? `SS?e?c"Kp??]uu?4Z?*?P?F0?$??????????.h??z?$"m? _a??aZ=?A|?i?p?h?????@5?Z?q?U?|???'??Q?? ??"?U???U Kaf???J??????002?%?-FI?f ?{???o??U|y?2B$?A?9v?!??F3<}?11U??dV?v#??B???|?I??f?q???(S??"?A???D ????O#????????:?0?[?p(S|?>V>kx???? ????G0??bM??9=?D????? 58?1?9?C/?d?Qd]???*??*K?>?$?z??u0s?0[????9?? ???'?=?}+k ????g?}?^#?HIK?Z k????X!??,??R ?^k ]???ogE?RCz=?4?7Y???"+?R V????????"&??? ?Y?????=R?ie?? ??r?TMe?EV?B?????J?JZb???j?T+?RYA?~? ?/?"(?`?"?jE' F(2J??d?Wf7?h??|?I?->1???'VjN??????(?"?$?:?tMB?~???????b[????QmH??b?~VT??E? Y??
??WY??J?B?B??0?
??\0$???
??\ ????(??*?? ?J?2??\ [???s??_e )??l????-?R?L??/???b?/f?7?G?[??>6?????sC?????#?-t??? `?:??????2pu?$D??(?i?CM?5???g?^- ?????T????m?????????&~lO7?7pY?nO;p??t????i?8.^??????u??L???????m??/f?=w?b?<?n?zw]????Apr?Nv???Yg?I?????U???b?|,?[?Ú?.n?7?????k?`?w6w?sm_??lN?n?j?V????????????5???kc?x;Z?j?h????zR?qOn???w?]?>W?][?<???rvV<?*?Q?3;:tN?u?? ?b[???zrX;~~i[?Fw??9???.???g???4??????????D????GO???Mo{??p??=??g[???t?9?^??y???vq?????%?j4?O7ow?7?????????{c_??w:???3?)oLz[???r??n?Ok{?+=??h????y8?^?z;j?X?.??F?~A???uw`????i{??j???I????YG#? l??O??^???&????9?-?>?O?[?tû~?lW???W}?]{?u?;???????v??v??W?????xc??F?n/????@9}-=?f????8??nqz???Y?t??????v?G??lL??Fy??N?? =?vq?q????Z??N???l?;jw?z{?=????n?7??.z?l?]nnVm???w??+??/??';??]????1?co?O?j?????N_?+?????JW????T???nw?2??v?????]?Y??5!c?????7???Y?v????3??+???????:*?h? ?^??\?????K???z??????g?QT??s??????FmX?V?o????Fcp?g?{?y>T????????J]????k?????????n{???m>]o??w+k?Y??.?k???l|2????i3{wt??;V??? ??= ? jIy6??2????8??????x?H?ht1e+? %???U?????? 3???^??? 7S???( ?-B???~???p? ???BX_??~?6Z??hT%?Y???_???????c?J???Y?*a?D?I>?9^'?qg??w ?2??2@o???@?L????2??????D? ?/??8?????b??<??m.J!$????A?B?>?p??g?????
[s(?]JAP?????/?????HxZdaG?gLC???B??U,?5??
?V1?QL#?-P'?Q 2?PL?*????2-???-??????????jgh2??x?k???H??=???? ?g-????$?B_???kh??-?\?PG????z9 ??%??H?_m ?*Fz???QX,gI?"k7u?&v??P??pn??????\?"k??Q??G??c?)=???=?4??,?"S )?H????j??d%?
??)??%?0$T???(??R?8???0 ??? ???t?3~?Fa3??%I ???*K???F|?g?M?W}?l?l?,???7<??j??[Oy? ?????U???b8h ??@o????8?pO?|??? ??bt%????? ???bqKSG????JK??5???M#~?
?D?ZS??,D??D
?>O???A??g??[???J
?F?s?!???j?]?S???E?+?i7?V??k??Zn?}~?,R? ??b3Q?;??3_1???YkA*7???z?(?I/???5C?k?^?#???<1xô??n??gj??kr??9?2??Sul?6?B??l3??:???G?5.? ?Y?;_?N?lU?m??TIYDM???????z??:?.??#?.F??$??*??O
]# ?J>0??????????eip???j?Z?si??????F??"OH+?d&???s??-D?)u?q?W7l;?Hg=?%?s}???Q3??gi??G??d????-U?????4S5?"Y???|?N??i???r???L??| ?OQ?|??M?w?H\??$??0"??????t?6????N$~?X?3n?}??|b8????H?<??[??0h;???????d???&??`??0??|p?1?*tw/e????8?v? nsY????N e????pK|K? >-???Q8%???PL??0>m??41???ZU?
*???Z?:9_/#3|!??o?J??D??a`?K?"??[4??Ii????? ?e6>
???b?4E?U?A?5????A,???i?~?+q??l?_2?b??Rf??P??6?o?M#?R"t?? ?_ ????C??(?t{?g??es???l???UB?92?;?@?? )2???'|??\.s?[V?HF|????????_2(?_????*?*? ?BFC??"w?? f????s???????BY??~ 6-*]? ?z?,M??+?+z)?^?1??
???\?#@??Y?u?a??@IwK^?^%???d?R??g1????_?V IDY?? ?Y?WH?C%? ?*Y1' GJJ [U??Rm??_OL?u???????G???Z)Rl 8u??m\`?,?????8Gx??0??+Q?;E???1e??2{:?=??S??????4??zb?(Q+;S???r??\???Z?;??{?p|??o0?j|?O?U????v???? r>???Z*7?D?JBr??F?z?D=?1&?$?R??`??v}?Q? ?e?/- `?X?K_q?I8?M??nt?"G ?RT#oxV*?[;?YWL?F?O?V*?WL??~???f?V2?'???????????#5?tV?"?C?? ?W??????k?(?y??_Z??{???g???H?MK+B??7?E??(??51~<?"?F??????4???K?}??q|C?bM*'??j?????l*:?r?Nn4?N???P`H???A<?k0vu-p?(??? ?i?!??qF?T*2B~v?R?b?*?%??@?q????[v??M%??*E??i??s??}??:??VU?]g??Kee ?Df>??_? :? ?g%??$????Gg??[??SD???*9O ?!?R?.$<???L???d??V$ ??tI/U $??_??}6H??8?& ?_?m??Q3??&??[??5(??@?f???X-???????D?@N??C??????8F?IH????^q?0|?(? ????D????W???H?jo???b??S(????wu' X~?W? e?6$?'?b/?_??????>?????S???QQ5?^W?|F??|F? _(&?J?Z??z?+??????5??f?d??? l $?7?I?e???q?!?E??S#??Ib??1?R ????rR????F?Sl??Q?Ve???H??S??L??p?G?? ?9?????????<?R???T}+??O???i?????q??g??Qd?tI?/???_????????d?Ut??w0???W??G???_7???_??????X`"??m?f!??ZT??Of|??$??8???f?? 7??
??vM1<??X-???? ?*??B???3?)D???pv9~E?d??!z? ?????C?????g½@??.?&??K??X>??#??R??H?&???_?a??M"?q???J%N?J?$+?U??^?r?[?zX?~v\1?E??]?Gs* ?b?gD?)')??{?H?l?Y?1? ????/????}???????p~?0ii?????N??c??c???/?/???? ?b?-??AL?5b???8Ew??????Ob?09W?<? ?F^?W?{?z?Y5C?Na?'?c< t{?<;?d??1d8?/1Hd????p?Atg|3[0????]Tf???W<?N?H?bE?fy!?K-???????#s?[??] >?}?wy????^^?W?H?k?[?I5??%????_ P????\?2um!???/oG??s??7w?}y?Y??f???4??w????U??l??

Any idea on how to decrypt the response body?  I have tried changing the protocol to sslv3 with no difference.
Just fyi there really aren't that many question marks but when I copied and pasted I guess a lot of special characters were lost.
« Last Edit: September 02, 2015, 11:34:47 pm by joer. »

Offline futzle

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3260
  • Karma: +192/-9
Re: lua https command response_body encrypted
« Reply #1 on: September 02, 2015, 11:36:35 pm »
Probably the response is gzip compressed, as one of the response headers indicates. Change your request to not ask for compression (gzip or deflate) and try again.

Offline joer.

  • Full Member
  • ***
  • Posts: 128
  • Karma: +1/-0
Re: lua https command response_body encrypted
« Reply #2 on: September 02, 2015, 11:43:01 pm »
Probably the response is gzip compressed, as one of the response headers indicates. Change your request to not ask for compression (gzip or deflate) and try again.

Wow.... duh.  Just by commenting out that accept-encoding line. I got my response.

Now to have fun and parse it :)

Thanks so much!

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: lua https command response_body encrypted
« Reply #3 on: August 18, 2018, 05:25:31 am »
Just stumbled across this and thought its pretty cool...

I changed the url used to a text only news site, just to see how the code works with more information - and can see the potential in grabbing content from those sorts of websites with Vera to e.g TTS news headline.