Detection of Silent SMS (part 2)
Detection of Silent SMS (part 2)
Created by: E3V3A
The original thread (#69 (closed)) was getting far too long, so this is the continuation of that thread.
Please use old thread as reference only!
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
Tasks
0Link issues together to show that they're related. Learn more.
Activity
- Author Contributor
Created by: E3V3A
@banjaxbanjo Actually my loglevel has been reset to normal. So I'm not high logging, but QMI and Xpose/XPrivacy is really spamming logcat very hard, especially when GPS and Wifi is on. My test phone is 2 core model, and I'm not sure continuously reading from or keeping buffer open is a good idea. Can you show me where in the code this is done?
- Author Contributor
Created by: SecUpwN
Please note: If we now close the other thread, it gets marked as solved, which will in turn trigger Bountysource to ask @agilob and me if the bounty (only $15 since noone gets their asses up and actually contributes anything, shame on you, app leechers) may be given to @banjaxbanjo. Continue.
- Author Contributor
Created by: ghost
@E3V3A here is the start of function https://github.com/SecUpwN/Android-IMSI-Catcher-Detector/blob/development/app/src/main/java/com/SecUpwN/AIMSICD/smsdetection/SmsDetector.java#L120-#L130
I've let run on my phone for 24 hours with no lag issue but with a bad cpu I suppose it might lag a little.
The main reason I kept logging open was duplicate sms but since we now check the timestamp this will no longer be an issue. Any ideas on what way to go?
- Author Contributor
Created by: E3V3A
So what do these thread sleepers do: here and here ??
I don't think it's the logcat dumper that takes any effort, it must be the checkForSilentSms() thread. Can we write it more efficiently? I see a lot of loops of loops, with lots of info shuffling. I'm sure we can make this more efficiently... But I need to think about and understand the code better.
Besides, you have better things to do... Let's leave this for now.
- Author Contributor
Created by: E3V3A
I'm wondering if there's some kind of indicator when a log buffer is getting full? (I.e. for the kernel to circle.)
- Author Contributor
Created by: ghost
@E3V3A I put in the sleep (1000) to slow down the logging abit so its not as laggy if you put this higher its basically like a timer anyway.
A lot of them loop you see are never access they are only accessed if a string is picked up.
- Author Contributor
Created by: E3V3A
ok, a few possible solutions:
- we can increase the size of the logcat buffer by piping to file, so we can make the sleep longer. (Is that blocking?)
- we can check the size before running.
- We might be able to increase performance by searching the logcat in binary format (-B), thus skipping the reformatting and i/o overhead.
# logcat -g -b radio -b main /dev/log/radio: ring buffer is 512Kb (17Kb consumed), max entry is 5120b, max payload is 4076b /dev/log/main: ring buffer is 512Kb (139Kb consumed), max entry is 5120b, max payload is 4076b
On my device, while running the app, it takes about 8 seconds to fill up
radio
buffer, and about 1m20s to fillmain
buffer, without doing anything!On more new devices with AOS > 4.3, the main buffer is 1 Mb.
But as I said before, I think its the string checking that takes the most time...
- Author Contributor
Created by: ghost
@E3V3A
But as I said before, I think its the string checking that takes the most time..
We need to check all strings I don't think there is a way around this and I have a check in place to only check small lines less than 250 chars and if string is not detected 90% of code is bypassed
if(progress[x].length() < 250){//check only short strings for faster processing
We can experiment with a filter but its going to be a very big filter and I think it will make detection not work as good and we will miss some data. With verbose logging all we have to check if the strings contains
Received short message type 0, Don't display or store it. Send Ack
and it doesn't matter what class this showed up in the logcat from, GsmInboundHandler etc.. < this works for other logcat basebands we dont know but as long as this string shows up it will alert no matter what Class the string came from. <<<Future proof
SmsDetector is not run on the UI thread its run on a background thread wont lag GUI so sleep wont cause issues I don't think.
- Author Contributor
Created by: E3V3A
A side note!
From the sources here:
/** * The WapPushManager service is implemented to process incoming * WAP Push messages and to maintain the Receiver Application/Application * ID mapping. The WapPushManager runs as a system service, and only the * WapPushManager can update the WAP Push message and Receiver Application * mapping (Application ID Table). When the device receives an SMS WAP Push * message, the WapPushManager looks up the Receiver Application name in * Application ID Table. If an application is found, the application is * launched using its full component name instead of broadcasting an implicit * Intent. If a Receiver Application is not found in the Application ID * Table or the WapPushManager returns a process-further value, the * telephony stack will process the message using existing message processing * flow, and broadcast an implicit Intent. */
- Author Contributor
Created by: E3V3A
Please re-test this issue now. (Using Buildozer build >= 211)
- Author Contributor
Created by: E3V3A
For some reason the SMS detection doesn't work in the latest build 227...
- Author Contributor
Created by: ghost
@E3V3A I'll do some testing adding the -b main makes it slightly longer to detect. I've tested wap and mwi recently and seem to be working ok will check again as well as type0.
- Author Contributor
Created by: E3V3A
I found a typo in the pre-loaded
DetectionStrings
table for Type-0 detection. It has a double ' in there. We can also shorten that string, to only use first sentence...current:
Received short message type 0, Don't display or store it. Send Ack
shortened:Received short message type 0
In addition I'd really like to see some
CB message
detection. - Author Contributor
Created by: SecUpwN
We can also shorten that string, to only use first sentence...
Not sure if that would lower detection probability of Type-0 SMS.
- Author Contributor
Created by: ghost
@SecUpwN no wouldn't make detection work less if anything it would work better. If the logcat line contains them sequence of characters then it will alert. Will add if I get the time and try change the db timestamps to int also.
- Author Contributor
Created by: SecUpwN
Will add if I get the time and try change the db timestamps to int also.
Thanks for clarifying @banjaxbanjo. Do yo ink you'll be able to pus your proposed changes within the next 2 hours? Not trying to be pushy or anything, but I'm preparing the release for our itchy users. ;-)
- Author Contributor
Created by: ghost
@SecUpwN no sec prob be tonight or tomorrow sometime
- Author Contributor
Created by: SecUpwN
no sec prob be tonight or tomorrow sometime
Perfectly fine, @banjaxbanjo. Make sure to use a fresh copy when you do. See ya later, alligator!
- Author Contributor
Created by: ga900
What will happen if regular SMS is sent with intentionally wrong checksum? Mobile will receieve and ack but then it will discard message as it was never receieved. Is there any way to detect such activity?
- Author Contributor
Created by: E3V3A
@ga900 That is a great question! Do you know how to write such an SMS we can use to test with?
SMS detection is still broken.. :(
@banjaxbanjo Can you please have a look what happened?
- Author Contributor
Created by: ga900
@E3V3A I do not know how to do it, but there are some rumours that such messages is better choice for E-OTD & UL-TOA measurements :( because type0 sms is visible on some multi-sim chinese phones.
- Author Contributor
Created by: TPS
I'm not sure if anyone else noticed, but the bounty on this bug (encompassing both #69 (closed) & #537) has now increased to $115 as of a couple weeks ago!
@SecUpwN Is there any way to get the bounty tag replicated from #69 (closed) so this issue shows (changes in) the bounty automatically as well?
- Author Contributor
Created by: SecUpwN
Is there any way to get the bounty tag replicated from #69 (closed) so this issue shows (changes in) the bounty automatically as well?
@TPS, thanks for asking this, I don't know. Maybe @rappo from @Bountysource does?
- Author Contributor
Created by: rappo
@SecUpwN @TPS The only way to have the bounty tag show up here automatically is if there's actually a bounty on this issue. If having the tag on this issue is important to you, you can still manually add / update it as needed.
- Author Contributor
Created by: SecUpwN
Thanks for responing, @rappo. As you can see, this is part 2 of the original Issue, but developers that continue here should get part of the huge bounty posted on the original Issue. Can I transfer some of it?
- Author Contributor
Created by: ghost
@SecUpwN @smarek Guys I think we can get this to work on non rooted phones less than api < 17 I haven't test yet but should work if @smarek wants to test somehow. if 17 or higher user will be asked for root comments added only for guide can be removed.
only needs permission.READ_LOGS if < api 17
try { Thread.sleep(500); Process process = null; String MODE = "logcat -v time -b radio -b main\n"; Runtime r = Runtime.getRuntime(); if(Build.VERSION.SDK_INT < 17){ process = r.exec(MODE); }else{ process = r.exec("su"); DataOutputStream dos = new DataOutputStream(process.getOutputStream()); dos.writeBytes(MODE); dos.flush(); dos.close(); } setSmsDetectionState(true); //<-- moved this here because we don't want to set true if we can't exec command. mLogcatReader = new BufferedReader(new InputStreamReader(process.getInputStream())); } catch (InterruptedException | IOException e) { Log.e(TAG, "Exception while initializing LogCat (time, radio, main) reader", e); }
- Author Contributor
Created by: smarek
@banjaxbanjo will test, thank you!
- Author Contributor
Created by: SecUpwN
@smarek and @banjaxbanjo, I've received an E-Mail from @DimaKoz with a VIDEO of a user testing AIMSICD and its capabilities of detecting silent SMS. Result: This detection needs more improvements.
- Author Contributor
Created by: ghost
@SecUpwN this is not a one for all app and is limited until we have more logs for other model phones. I've shown app detecting all sms on my Samsung S5 when debugging set to high in menu *#9900#.
The success of this is based on the amount of logs that we receive with users receiving a silent sms.
so far I've got it to this stage with only my own logs so would be great to get other unique strings to indicate a type 0 etc..
We don't want user data all we need is a string that we know is indicating a silent sms.
- Author Contributor
Created by: SecUpwN
@CellularPrivacy/silentsms would you please be so kind to continue improving our detection mechanisms? Our app currently seems to not detect any silent SMS, at least no Self-Test works.
- Author Contributor
Created by: smarek
Also, Silent SMS Ping could be tested and detected. The message is from time to time logged in radio syslog while being thrown out (phone doesn't know how to process the message further).
- Author Contributor
Created by: ghost
@SecUpwN @smarek Sms detection will not work as it stands in this repo. The detection will never get called unless this is changed
// as soon as logcatLines.size is greater than buffer lines will keep on getting appended //so the else block will never be called (where dection is done) if (logcatLines.size() >= LOGCAT_BUFFER_MAX_SIZE || logcatLine != null) {
logcatLines.add(logcatLine);
}
//needs to be if (logcatLines.size() <= LOGCAT_BUFFER_MAX_SIZE || logcatLine != null) {
logcatLines.add(logcatLine);
}
I've explained this before but got no reply. Look at SmsDetector on my repo on how long this issue was there, fix is also there.
As smarek said before the class has bad OOD and needs overhaul which I cannot do atm because of time constraints. Will do full recode when have time. (2+ months)
- Author Contributor
Created by: smarek
@banjaxbanjo i've created pull request
- Author Contributor
Created by: SecUpwN
Thank you so much for having another look, @banjaxbanjo! Did not mean to ignore your replies. Also thank you for creating the pull request, @smarek. Sadly though, still no Self-Tests are working.
- Author Contributor
Created by: smarek
Well, i've taken another look, do I understand correctly, that we're doing detection based on detecting these lines in LogCat ?
Received short message type 0 Received voice mail indicator clear SMS shouldStore=false SMS TP-PID:0 data coding scheme: 24 isTypeZero=true incoming msg. Mti 0 ProtocolID 0 DCS 0x04 class -1 SMS TP-PID:0 data coding scheme: 4
If so, I wonder, what we expect to be the outcome, as these messages never appeared on my test devices.
What I can see though is ie. this:
W/SmsMessage( 1044): 1 - Unsupported SMS data coding scheme 4 W/SmsMessage( 1044): 1 - Unsupported SMS data coding scheme 4
When receiving silent sms
Would it be possible to run tests and provide LogCat output for you @SecUpwN ? Indeed would be great to have enough data to search for. Interesting keywords being RIL (RILC/RILJ/...), SMS (in various letter cases) and GSM/CDMA (also because of tagging)
- Author Contributor
Created by: SecUpwN
Well, i've taken another look, do I understand correctly, that we're doing detection based on detecting these lines in LogCat?
Currently, yes. @banjaxbanjo implemented testing of strings entered in
Settings/Detection Strings
.If so, I wonder, what we expect to be the outcome, as these messages never appeared on my test devices.
That is the drawback we currently have, not all devices support setting verbose log outputs.
Would it be possible to run tests and provide LogCat output for you @SecUpwN?
Yes. Please tell me which tests I shall run on my rooted device. Thanks for continuing with our journey!
- Author Contributor
Created by: smarek
If you could run individual sms types from Self-Tests against device where you do watch the logcat and post the block, which is the LogCat output around the sms receive along with note, which type of sms does this block belong to.
Also we're talking about verbose logs, so something linke this
adb logcat -b radio -b system -b events -b main
- Author Contributor
Created by: SecUpwN
post the block, which is the LogCat output around the sms receive along with note, which type of sms does this block belong to
Yes, I'll do that. I'm currently on the jump and will tackle this tomorrow. For wh<at do I have to look in specific (e .g.
W/SmsMessage
)? Is there a way to only filter the important messages to make sure? - Author Contributor
Created by: smarek
@SecUpwN
Interesting keywords being RIL (RILC/RILJ/...), SMS (in various letter cases) and GSM/CDMA (also because of tagging)
Also I propose to add new default detection string:
Unsupported SMS data coding scheme
Where we're interested in various coding schemes, that may indicate Wap Push or Type0
- Author Contributor
Created by: SecUpwN
Unsupported SMS data coding scheme
What would be the
Type
for thatString
? Please craft a pull request for it. - Author Contributor
Created by: smarek
There is no type in this detection, it's SMS in for.at not known to thr UAP (OS, Android). So if you need one, call it
Suspicious
- Author Contributor
Created by: SecUpwN
Interesting keywords being RIL (RILC/RILJ/...), SMS (in various letter cases) and GSM/CDMA (also because of tagging)
@smarek, I really would love to help on improving our SMS Detection and finally getting this to work to a somewhere stable state so that even unrooted phones can make use of this. Thing is, I get huge logfiles when testing each of the SMS types from
HushSMS
. So what I essentially need, would be a way to erase my private data likeCID
,LAC
and other identifyable data so that I could share logfilkes with you. If we had a way to enable our users to contribute such data without risking their privacy, I'm sure we could heavily improve our SMS Detection (that is why we filed #491 as a first step). Ok, here are my results, I hope @CellularPrivacy/silentsms will chime in to improve the existing codebase of our SMS detection:PING
- Let HushSMS choose Method (most recommended option)
04-09 05:38:38.611 1423 1550 I Xposed : HushSMS: <- NSMe: Cant get msg class: java.lang.NoSuchMethodError: No virtual method getMessageClass()Landroid/telephony/SmsMessage$MessageClass; in class Lcom/android/internal/telephony/gsm/SmsMessage; or its super classes (declaration of 'com.android.internal.telephony.gsm.SmsMessage' appears in /system/framework/telephony-common.jar) 04-09 05:38:38.612 1423 1550 I Xposed : HushSMS: <- Do we have a message class?: UNKNOWN 04-09 05:38:38.622 1423 1550 D GsmInboundSmsHandler: Received short message type 0, Don't display or store it. Send Ack 04-09 05:39:01.296 1423 1550 D RILJ : [5884]> SMS_ACKNOWLEDGE true 0 [SUB0]
Result: Our app did not recognize it, not even after fully expanding the detection string
Received short message type 0
that we already use for the silent SMS. Nothing stores in tabDelete Detected SMS
.
- Use Silent Ping (Type0) (Don't use this if you are on the German Telekom network)
04-09 06:02:19.667 1423 1423 I Xposed : HushSMS: -> persistMessage: allyourmessagesarebelongtous|1|0 04-09 06:02:19.667 1423 1423 I Xposed : HushSMS: -> handleMessage: success 04-09 06:02:19.670 1423 1550 I Xposed : HushSMS: <- dispatchMessage: Message is NOT null. Let's take a look. 04-09 06:02:19.670 1423 1550 I Xposed : HushSMS: <- dispatchMessage: Here's what I know: <phone number removed> 04-09 06:02:19.671 1423 1423 D MmsService: getAutoPersisting 04-09 06:02:19.672 1423 1550 I Xposed : HushSMS: <- NSMe: Cant get msg class: java.lang.NoSuchMethodError: No virtual method getMessageClass()Landroid/telephony/SmsMessage$MessageClass; in class Lcom/android/internal/telephony/gsm/SmsMessage; or its super classes (declaration of 'com.android.internal.telephony.gsm.SmsMessage' appears in /system/framework/telephony-common.jar) 04-09 06:02:19.672 1423 1550 I Xposed : HushSMS: <- Do we have a message class?: UNKNOWN 04-09 06:02:19.676 1423 1550 D GsmInboundSmsHandler: Received short message type 0, Don't display or store it. Send Ack 04-09 06:02:19.676 1423 1550 D RILJ : [5916]> SMS_ACKNOWLEDGE true 0 [SUB0]
Result: Same as previously, no detection at all.
WAP PUSH
- WAP PUSH SI
04-09 06:23:23.583 1423 1550 I Xposed : HushSMS: <- NSMe: Cant get msg class: java.lang.NoSuchMethodError: No virtual method getMessageClass()Landroid/telephony/SmsMessage$MessageClass; in class Lcom/android/internal/telephony/gsm/SmsMessage; or its super classes (declaration of 'com.android.internal.telephony.gsm.SmsMessage' appears in /system/framework/telephony-common.jar) 04-09 06:23:23.583 1423 1550 I Xposed : HushSMS: <- Do we have a message class?: UNKNOWN 04-09 06:23:23.597 16453 16453 E HushSMS:: Received intent. 04-09 06:23:23.600 1423 1550 V BlacklistProvider: Query uri=content://blacklist/bynumber/%2B<phone_number_removed>, match=2 04-09 06:23:23.609 1423 1550 D GsmInboundSmsHandler: destination port: 2948 04-09 06:23:23.645 1423 1550 D GsmInboundSmsHandler: URI of new row -> content://raw/1 04-09 06:23:23.646 1423 1550 D RILJ : [7029]> SMS_ACKNOWLEDGE true 0 [SUB0] 04-09 06:23:23.647 1423 1550 D GsmInboundSmsHandler: DeliveringState.processMessage:2 04-09 06:23:23.647 223 290 D QC-QMI : Getting maximum message length 04-09 06:23:23.647 223 290 D QC-QMI : Encoding buffer 04-09 06:23:23.647 223 290 D QC-QMI : Setting up transaction 04-09 06:23:23.647 223 290 D QC-QMI : Sending message 04-09 06:23:23.647 223 290 D QC-QMI : Releasing Transaction 04-09 06:23:23.648 1423 1550 W SmsMessage: 1 - Unsupported SMS data coding scheme 4
Result: This was actually a bit interesting. Even though I could not find significant things in the log, it appears that
AIMSICD
switched to Status: Medium and vibrated, displayingOpenCellID not found in Database!
when the WAP Push had been sent. @banjaxbanjo, could it be that our app indeed detects the message, but fires the wrong alarm and also no popup like it should be?
- WAP PUSH SL
04-09 06:39:40.635 1423 1550 I Xposed : HushSMS: <- Do we have a message class?: UNKNOWN 04-09 06:39:40.641 16453 16453 E HushSMS:: Received intent. 04-09 06:39:40.645 1423 1550 V BlacklistProvider: Query uri=content://blacklist/bynumber/%2B<phone_number_removed>, match=2 04-09 06:39:40.654 1423 1550 D GsmInboundSmsHandler: destination port: 2948 04-09 06:39:40.668 223 2124 D QC-QMI : Calling cooked async Callback 04-09 06:39:40.668 223 2124 D QC-QMI : Inside qmi_client_decode_msg_async Callback 04-09 06:39:40.668 1423 1516 D RILJ : [7691]< SEND_SMS { mMessageRef = 178, mErrorCode = -1, mAckPdu = null} [SUB0] 04-09 06:39:40.669 1423 1423 I Xposed : HushSMS: -> handleMessage() was called 04-09 06:39:40.669 1423 1423 I Xposed : HushSMS: -> handleMessage: success 04-09 06:39:40.698 1423 1550 D GsmInboundSmsHandler: URI of new row -> content://raw/1 04-09 06:39:40.698 1423 1550 D RILJ : [7704]> SMS_ACKNOWLEDGE true 0 [SUB0]
Result: Same as with sending a
WAP Push SI
, our app shortly displays a wrong status and vibrates.
- CLASS 0
04-09 06:34:45.701 1423 1550 I Xposed : HushSMS: <- Do we have a message class?: CLASS_0 04-09 06:34:45.705 16453 16453 E HushSMS:: Received intent. 04-09 06:34:45.712 1423 1550 V BlacklistProvider: Query uri=content://blacklist/bynumber/%<phone_number_removed>, match=2 04-09 06:34:45.758 1423 1550 D GsmInboundSmsHandler: URI of new row -> content://raw/1 04-09 06:34:45.758 1423 1550 D RILJ : [7326]> SMS_ACKNOWLEDGE true 0 [SUB0] 04-09 06:34:45.760 1423 1550 D GsmInboundSmsHandler: DeliveringState.processMessage:2 04-09 06:34:45.761 223 290 D QC-QMI : Getting maximum message length 04-09 06:34:45.761 223 290 D QC-QMI : Encoding buffer 04-09 06:34:45.761 223 290 D QC-QMI : Setting up transaction 04-09 06:34:45.761 223 290 D QC-QMI : Sending message 04-09 06:34:45.761 223 290 D QC-QMI : Releasing Transaction
Result: The
Class 0
messages where delivered to my inbox, but no reaction ofAIMSICD
at all.- MWI Message
04-09 06:51:40.790 1423 1550 D GsmInboundSmsHandler: Received voice mail indicator clear SMS shouldStore=false
Result: Interestingly
AIMSICD
vibrates and switches toStatus: Medium
again while complainingCell ID not OpenCellID Database!
. Here is what I personally suspect: We currently have an error somewhere in our code where our app indeed does detect some special messages, but not fire the correct warning.@smarek, as you can see from the few log snippets I attached, your requested default detection string
Unsupported SMS data coding scheme
also appears in logs of the other SMS classes. How do you plan to prevent a messup of detections if we add it? I already tested detections after adding it: No new results. - Author Contributor
Created by: Ddedalous
is this project still current