SjASMPlus от z00m

[Ped7g]

https://github.com/z00m128/sjasmplus...es/tag/v1.18.2

• new exist operator to check label existence
• the --syntax=i mode makes now also register parsing case insensitive
• minor bugfixes (predefined values, savenex BMP loader less strict about "colors used" content)

[Dart Alver]

Мне одному кажется что эту тему давно следовало бы пришпилить в "Важно:" , наравне с "SjASMPlus Z80 кросс ассемблер" или даже как более актуальную ?

[Shadow Maker]

@Ped7g as a Linux user I am very interested in converting characters in "ENCODING" from UTF-8 to one of these (WIN/DOS). But unfortunately it does not work it seems. Is it possible to implement? Or maybe I am missing something here. Actually it would be nice to have a full-fledged iconv or similar, as now it is limited to russian encodings...

- - - Добавлено - - -

Мне одному кажется что эту тему давно следовало бы пришпилить в "Важно:" , наравне с "SjASMPlus Z80 кросс ассемблер" или даже как более актуальную ?

Да, готово. Просто ее постоянно поднимали, я и не заметил.

[Ped7g]

@Ped7g as a Linux user I am very interested in converting characters in "ENCODING" from UTF-8 to one of these (WIN/DOS). But unfortunately it does not work it seems. Is it possible to implement? Or maybe I am missing something here. Actually it would be nice to have a full-fledged iconv or similar, as now it is limited to russian encodings...

I'm also on linux and used to utf-8 everywhere, so I do get your idea, but I'm not sure what/how to add to sjasmplus.

As ZX asm programmer I expect everything to be 8bit, so the utf8 -> 8bit conversion makes sense, but by what rules? In our ZX demos we often use custom encoding, not even some old DOS or win CP page, but completely custom one, so no generic tool will help much with that.

Maybe with Russian texts you have less mixing and use only few encodings, but I'm still not sure how to define one.

And finally the sjasmplus project currently doesn't contain any utf-8 implementation, so if I would link against iconv/similar, it will grow the dependencies list, so it should be rather something very well working feature, to make that cost worth it.

BTW if you need some standard win/dos encoding, I guess you can add into your makefile/build script the pre-build step using iconv itself, for example having .asm in utf8, and "building" .a80 files by implicit makefile rule using iconv similar to this:

$ echo "Мне одному кажется" | iconv -f UTF-8 -t CP1251 | hd
00000000  cc ed e5 20 ee e4 ed ee  ec f3 20 ea e0 e6 e5 f2  |... ...... .....|
00000010  f1 ff 0a                                          |...|

and then build the ".a80" with sjasmplus. If you use the utf-8 chars only within double quotes or comments, it should work.

So under linux, for general conversion like utf-8 -> cp1251 I don't feel sjasmplus needs any change, you can easily work around that (I have no idea if other OS have iconv and other powerful tools, maybe it's more difficult on other systems).

But if you guys have some cool ideas, how to bring into sjasmplus something even better, something what would do even things which iconv can't cover and|or makes life easier for people who use completely custom encoding, let me know.

But I can't imagine any particular nice syntax for some new directive covering these special cases, and when I recently was helping on one sjasmplus project which was "scrambling" text strings with custom xor-scheme to make strings hidden from simple view, I did end writing macro using the {b adr} memory read, changing the regular `db "some text"` into final scrambled bytes in DUP-loop produced by encoding macro. So even many of custom encodings (following some simple formula for 90% of chars and having only few special rules) could be done quite easily in sjasmplus with post-process macros.

Feels to me like it's not very difficult to resolve any of this use-cases even with current sjasmplus, but I have difficult time to imagine change which would help and be also elegant and worth implementing. (except the obvious utf8->cp1251 internal conversion, but that feels to me a bit useless, as I can use `iconv` for that already).

- - - Updated - - -

edit: to make that command line more complete, including the sjasmplus... (just for my own amusement and test)

$ echo "txt: db \"Мне одному кажется\"" | iconv -f UTF-8 -t WINDOWS-1251 | sjasmplus - --raw=- | hd
SjASMPlus Z80 Cross-Assembler v1.18.2 (https://github.com/z00m128/sjasmplus)
Pass 1 complete (0 errors)
Pass 2 complete (0 errors)
Pass 3 complete
Errors: 0, warnings: 0, compiled: 2 lines, work time: 0.001 seconds
00000000  cc ed e5 20 ee e4 ed ee  ec f3 20 ea e0 e6 e5 f2  |... ...... .....|
00000010  f1 ff                                             |..|

[Shadow Maker]

I'm also on linux and used to utf-8 everywhere, so I do get your idea, but I'm not sure what/how to add to sjasmplus.

Thanks for your detailed reply Yes, sure, I know how to do it without sjasmplus, but I feel shame that we have commands which are not really usable
In russian texts we usually use native dos or win russian encoding, custom encoding is also used, but it is not really common nowadays, when everyone have access to PC and code with sjasm, why invent something else
Only idea I have then is custom table approach (so basically a replacement list), so have it like:

replace_start "table1"
    db "тест тест"
replace_end

or maybe this (but not sure what best way would be to select individual lines, so I added as a comment)

    db "тест тест" ;replace "table1"

but not sure if it is really usable.

[Ped7g]

I'm not sure what you mean by those proposals.

Current sjasmplus reads source code in binary 8-bit mode, so whatever is inside double quotes in DB except few control codes which need to be escaped (`\0\n`) will be 1:1 assembled into machine code (I should probably do the test doing full 0..255 char string to be sure it works like that, yeah, I will add one).

So as ZX SW author, you have some encoding on mind (CP1251 or DOS-866 or some custom like 131 = "star" and 132-133-134 "group logo"), and you need to put those values (CP1251 azbuka chars or 131..134 bytes) into machine code, usually DB statement.

That's the result side. And the way how you edit those texts, for example Russian strings, is nice with UTF8 because of modern text editors using utf8 by default.

Current sjasmplus has this ENCODING directive, which makes possible auto-conversion from cp1251 to DOS-866 - it does nothing else, by default it does nothing, and if you do `ENCODING DOS` or use `--dos866` CLI option, any 128+ byte value in source code is transformed by hard-coded table converting CP1251 to DOS-866 (or "damaging" anything else, like UTF8).

So in case of Russian string, the task is "simple", do the utf8 -> cp1251 (or DOS-866) of source before the DB is assembled.

(BTW let's make clear one thing - I'm not going to add UTF8 support for symbol names, so anything outside of quotes does not need any extra support by sjasmplus, as anything non-ASCII is bug in source - the reasons are mostly overlapping with what I will write below, plus extra pain of sjasmplus processing source code heavily with custom implementation, not re-using common C++ library, so utf-8 symbols would cause major rewrite of parser - if somebody wants to do that, ok, but not me, the benefit of unreadable symbol names doesn't attract me at all, even if the code change would be simple)

But this "simple" case means the sjasmplus code will have to learn utf-8, and have the conversion table. And if there is Russian table, why not to add also some german, czech, etc...
... and you end up implementing `iconv` - which is by no means simple task, implementing utf-8 support correctly is major pain, I know of one C++ framework avoiding iconv and using custom code, and it took few years to polish that implementation enough to mostly work as expected.

And at that moment I don't see any benefit of putting iconv into sjasmplus, if I can call the iconv externally as I have shown in that example in previous post.

I can see some benefit of some magic directive which would allow me to define custom encoding, ie. that ★ is 130 and ☈☋☑ is 131,132,134, but I don't see any elegant and symple syntax for that, and the implementation again requires adding all the important parts of what iconv does, ie. understanding utf-8 encoding correctly.

So if you want just the "simple" utf-8 to cp1251 or dos-866 conversion, I'm failing to see why to bloat the sjasmplus code, and not call the external `iconv` as intermediate step before assembling. The result is same, but iconv is more robust and could handle all the common encodings, while sjasmplus will be always very limited in what it knows, unless I re-implement whole iconv into it (and go from ~300kB binary to ~5MB assembler).

- - - Updated - - -

So, I added the test to verify that "anything 8bit inside quotes (except the sensitive control codes) works":
https://github.com/z00m128/sjasmplus...t_encoding.asm

The "sensitive control codes" contains three values: 0, 10 and 13 ("\0\n\r" escape sequences within double-quotes).

All the other 0..255 values are assembled 1:1 to machine code.

So this part of sjasmplus works "as intended" and there's no extra bug involved or any issue.

The [utf8 text source] -> [8bit source] conversion is IMO lot easier to handle externally, and I'm slightly against adding this functionality into sjasmplus (you can of course try to change my mind, but I don't see enough arguments at this moment).

I can see the convenience of such addition, but considering the current size of sjasmplus code and its build-dependencies, I find it not worth of adding utf-8 support, especially as the external usage of `iconv` is trivial in case of non-custom encoding, and does cover LOT MORE than just Russian encodings.

Maybe on non-linux OS the benefit of built-in conversion would be even bigger (if they don't have easy tool like `iconv`), but then again, it's lot more easier to install linux and use it for assembling ZX project, than to modify sjasmplus sources, so my general advice is to use modern OS, and not to reinvent the wheel again and again inside sjasmplus just because some other SW is obsolete.

- - - Updated - - -

One more note... I was even going to propose to call `iconv` from the source with SHELLEXEC (to convert some small "strings.asm" and then INCLUDE the converted one), but it turns out that's not so easy. The SHELLEXEC does execute only in the last third pass, so the converted strings are not available in earlier passes. I guess you can still do this in lua-script, calling `iconv` in first pass to generate "strings.8b.asm" (cp1251 encoded) from "string.asm" (utf8 - what you edit in editor), and then INCLUDE the converted file - let me know if you need example how to do this.

But I generally prefer to not use lua scripts in my asm, so I would instead rather create Makefile with the rule to produce that converted file before assembling of main project.

[Advertiser]

[Bedazzle]

What about some kind of "translate" command, where you define two strings

source = "АБЦДЕФГ....."
target = "ABCDEFG...."

And later in code

call print_string
translate("АБЦД 123")

Sjasm is going to translate string char by char, and does put same symbol if it is not found in translator strings (like for 123 in example)?

[Ped7g]

Doesn't help that much until you implement utf-8 parsing. It could cover custom encodings 8bit -> 8bit, but then your syntax doesn't explain how you will define for example A -> 1, B -> 2, etc.. values which are not easy to enter into quotes.

Thinking about it, it's like two different issues. One is "utf-8 anything", and my answer is "no", I don't see how to add utf-8 support to sjasmplus without either adding lot of own code, or linking against some ICU-like library, but in either case raising the complexity and size of sjasmplus binary by whole order. And you can resolve the utf-8 by simply converting the source with `iconv` in the build script to some 8bit classic encoding, which then is assembled by sjasmplus correctly (I don't see anything problematic about this external way).

Second issue is "custom 8bit encoding" - I had to resolve few of these in my own ZX projects, and usually I enter the text in numbers or post-process the data by script written in sjasmplus macro. If the conversion would be even lot more complex, you can always do something very similar to what you propose with "translate(...)" in lua. So the status on this one is, that you can resolve it in current sjasmplus, but if somebody shows me more elegant syntax (to define custom encoding), I may implement that. Right now all the syntax I can imagine for such feature doesn't feel very attractive - I would have to study the docs before using it any way, to use it correctly, and in such case I could probably in similar time write the post-process macro changing the values in classic way in script code.

It just doesn't feel like I can add to sjasmplus something meaningful, what will help in most of the use cases and be easy to use, feels like I can add something what will work well for specific use-case, but will be mostly ignored by everyone else. Also I don't remember some nice solution from other assemblers to just copy it.

[NEO SPECTRUMAN]

тот же bug что и в define

	macro coord_x x

var_x 		di : halt
data_x		nop
sdfgj_x = var_x

	endm
	
	
	org $8000
	
	coord_x ($5+1)

test.asm(8): error: Invalid labelname: var_($5
test.asm(17): ^ emitted from here
test.asm(8): error: Unrecognized instruction: ) 		di
test.asm(17): ^ emitted from here
test.asm(8): error: Unexpected: ) 		di
test.asm(17): ^ emitted from here
test.asm(9): error: Invalid labelname: data_($5
test.asm(17): ^ emitted from here
test.asm(9): error: Unrecognized instruction: )		nop
test.asm(17): ^ emitted from here
test.asm(9): error: Unexpected: )		nop
test.asm(17): ^ emitted from here
test.asm(10): error: Invalid labelname: sdfgj_($5
test.asm(17): ^ emitted from here
test.asm(10): error: Unrecognized instruction: ) = var_($5+1)
test.asm(17): ^ emitted from here
test.asm(10): error: Unexpected: ) = var_($5+1)
test.asm(17): ^ emitted from here

17    8000              	coord_x ($5+1)
17    8000             >
test.asm(8): error: Invalid labelname: var_($5
test.asm(17): ^ emitted from here
test.asm(8): error: Unrecognized instruction: ) 		di
test.asm(17): ^ emitted from here
test.asm(8): error: Unexpected: ) 		di
test.asm(17): ^ emitted from here
17    8000             >var_($5+1) 		di
17    8000 76          >  halt
test.asm(9): error: Invalid labelname: data_($5
test.asm(17): ^ emitted from here
test.asm(9): error: Unrecognized instruction: )		nop
test.asm(17): ^ emitted from here
test.asm(9): error: Unexpected: )		nop
test.asm(17): ^ emitted from here
17    8001             >data_($5+1)		nop
test.asm(10): error: Invalid labelname: sdfgj_($5
test.asm(17): ^ emitted from here
test.asm(10): error: Unrecognized instruction: ) = var_($5+1)
test.asm(17): ^ emitted from here
test.asm(10): error: Unexpected: ) = var_($5+1)
test.asm(17): ^ emitted from here
17    8001             >sdfgj_($5+1) = var_($5+1)
17    8001             >

и если define это можно простить
то для macro это вообще недопустимое поведение


хотя отдельный replaceallmacro именно с этим же функционалом не помешал бы

кстати по ходу это уже давно

sjasmplus-1.11.0

# file opened: test.asm
test.asm(1): error: Invalid labelname: 

 6    0000              	macro mcr x
 7    0000 ~
 8    0000 ~            label_x = 1
 9    0000 ~
10    0000              	endm
11    0000
15    0000              	org $8000
16    8000
17    8000              	mcr 4
17    8000             >
17    8000             >label_x = 1
17    8000             >

sjasmplus-1.12.0+

 6    0000              	macro mcr x
 7    0000 ~
 8    0000 ~            label_x = 1
 9    0000 ~
10    0000              	endm
11    0000
15    0000              	org $8000
16    8000
17    8000              	mcr 4
17    8000             >
17    8000             >label_4 = 1
17    8000             >

[Ped7g]

yes, the 1.11.0 (and older) had inconsistent behaviour, sometimes substituting `label_x` with "x" macro argument, sometimes not (I don't remember exact details how to trigger it, but it was fairly trivial to modify your example a tiny bit, and it would start substituting the `label_x` also in 1.11.0 ... IIRC all it takes is to have for example another label: `xmax` which doesn't get substituted, but will affect `label_x` ... or something like that... it's now two years since I fixed it, I would have to check the old code to be sure how to trigger the old bug.

The 1.12+ does consistently substitute sub-word (every time) - what you see is "fix".

To fix your source in later one, don't use trivial argument names like `x` ... I personally suggest `x?` for macro arguments, or you can add underscore to prevent mid-word substitution like `_x` for macro argument.

Unfortunately the current state is based on huge misunderstanding. The original patch (to one of 1.07 RC versions I think) had bug, causing it to substitute `x` also in `label_x` when certain conditions were met (in the define-hashtable, collision on the first letter, affecting size of "bucket" and making the "x" found even in case it should have been ignored). And there was test in old test suite, testing the bugged behaviour!

So initially I was confused by the inconsistency, and I fixed it, to make the substitution to work always, but I fixed it the way how the old test was verifying it. And the defines/macro-args starting with underscore can substitute only at beginning of identifier, not in the middle.

Few versions later I finally understood how the patch was originally meant, it was supposed to do sub-word substitution in the opposite way, only with identifiers which do start with underscore. But unfortunately the original author of the patch didn't put any comments into the code, and didn't provide any tests, and later somebody added the wrong test testing the bugged behaviour. If the original author would document his idea, I would fix it the correct way.